Audio augmented reality cues to help you focus on audible information

Abstract

An audible augmented reality (AR) computing system may receive audio data, such as audible information, ambient noise in an operating room (OR), from a sensing system in an OR. The audio AR computing system may generate AR content. For example, the audio AR computing system may generate AR content based on the audible data. The AR content may be or include audible information associated with the audio data. The audio AR computing system may receive adjustment instructions. The adjustment instructions may be or include one or more of adjustment information for the AR content. For example, the adjustment instructions may be or include one or more of a surgical task instruction, a task importance instruction, a voice insertion instruction, a voice conversion instruction, and a sound source location instruction. The audio AR computing system may adjust the AR content based on the received adjustment instructions.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is related to the following concurrently filed applications, the contents of each of which are incorporated herein by reference: U.S. Patent Application No. 17 / 156,287, entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER," attorney docket number END9290USNP1. [Background technology]

[0002] Surgical procedures are typically performed in a surgical site or operating room within a medical facility, such as a hospital. A variety of surgical devices and systems, and / or sensing systems are utilized in performing surgical procedures. In the digital and information age, medical professionals can utilize technology to improve patient care. It would be desirable to find ways for medical professionals to utilize augmented reality (AR) computing systems, such as audio and / or visual AR computing systems, to improve patient care. Summary of the Invention [Means for solving the problem]

[0003] An audio augmented reality (AR) computing system can include a processor configured to receive audio data from a sensing system in an operating room, the audio data including measurement data, generate AR content based on the received audio data, obtain or receive adjustment instructions indicating adjustment information for the AR content, and adjust the generated AR content based on the adjustment instructions.

[0004] The audio data, including the measurement data, may be associated with a user, such as a medical professional, an HCP (e.g., a surgeon), or a patient. Advantageously, audible information associated with particular measurement data that is relevant and / or important to the current surgical procedure may be adjusted or amplified (e.g., increased in volume). Accordingly, perception of the surgical procedure, the atmosphere in the OR, and / or interaction between HCPs may be improved.

[0005] The adjustment instructions may be received from the surgical computing system and may include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a voice source location instruction indicating the voice source location of the voice data.

[0006] The adjustment instructions may include a significance of the surgical step, and the processor may be further configured to identify an audio AR setting associated with the significance of the surgical step and adjust the AR content according to the audio AR setting. Advantageously, the AR content may be adjusted according to the significance of the surgical step, which may improve perception of the surgical procedure.

[0007] The adjustment instructions may optionally include audio information for critical surgical steps, and to adjust the generated AR content, the processor may be configured to mute audio data from the sensing system and optionally amplify audio associated with the audio information for the critical surgical steps. Advantageously, the audibility of audio information that may be related to critical surgical steps may be increased, improving awareness of the surgical procedure.

[0008] The audio information for the critical surgical step may further include at least one of an increase frequency instruction or a decrease frequency instruction, and the processor may be configured to: send an increase frequency request to the sensing system based on the audio information including the increase frequency instruction, the increase frequency request including a request to increase the frequency of transmitting audio data from the sensing system; or send a decrease frequency request to the sensing system based on the audio information including the decrease frequency instruction, the decrease frequency request including a request to decrease the frequency of transmitting audio data from the sensing system. Advantageously, audibility of audio information that may be associated with critical surgical steps may be increased, improving awareness of the surgical procedure.

[0009] The adjustment instructions may include surgical task instructions indicating a surgical task being performed or to be executed, and the processor may be configured to identify relevance of audio data from the sensing system to the surgical task indicated in the surgical task instructions and, based on the identified relevance to the surgical task indicated in the surgical task instructions, determine whether to block the audio data from the sensing system, where the audio data from the sensing system is blocked if the audio data from the sensing system is unrelated to the surgical task indicated in the surgical task instructions and / or the audio data from the sensing system is allowed if the audio data from the sensing system is associated with the surgical task indicated in the surgical task instructions. Advantageously, the HCP may, for example, only attend to audio data related to the surgical task being performed or to be executed. Accordingly, awareness of the surgical procedure may be improved.

[0010] The processor may be configured to receive an ambient noise level indication indicating an ambient noise level in the operating room, and, on a condition that the received ambient noise level is below a threshold ambient noise level, send a critical task indication to the surgical computing system, the critical task indication indicating that a critical surgical task should be performed. Advantageously, the system may determine that the HCP is about to perform a critical task and may send an alert to other HCPs that an upcoming task involves a critical surgical task.

[0011] Prior to adjusting the generated AR content, the processor may be configured to: send a user input request to the surgical computing system, the user input request requesting user input; and receive user input associated with the user input request, the AR content being further adjusted based on the user input. Advantageously, the user can tailor or adjust the AR content.

[0012] The adjustment instructions may include surgical task instructions indicating a surgical task being performed or to be performed, and the processor may be configured to identify an audio AR setting associated with the surgical task and adjust the AR content according to the identified audio AR setting. Advantageously, the AR content may be adjusted according to the surgical task being performed or to be performed. Thus, perception of the surgical procedure may be improved.

[0013] The audio data may include first audio data, the sensing system may comprise the first sensing system, the adjustment instructions may include user preference settings associated with the surgical procedure, and the processor may be configured to: receive second audio data from a second sensing system in the operating room, select preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on the user preference settings, and adjust the AR content by increasing a volume of the selected preferred audio data. Advantageously, the system can handle multiple audio data sources according to user preferences.

[0014] The audio data may include first audio data, the sensing system may include the first sensing system, the adjustment instructions may include user preference settings associated with the surgical procedure, and the processor may be configured to: receive second audio data from a second sensing system in the operating room, select preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on the user preference settings, and adjust the AR content by reducing a volume of the first audio data from the first sensing system if the second audio data from the second sensing system is selected as the preferred audio data. Advantageously, the system can handle multiple audio data sources according to user preferences.

[0015] The audio data may include first audio data, the sensing system may comprise the first sensing system, the adjustment instructions may include user preference settings associated with the surgical procedure, and the processor may be configured to receive second audio data from a second sensing system in the operating room, and, on a condition that the first audio data is preferred based on the user preference settings, perform at least one of: sending a first frequency increase request to the first sensing system to increase a frequency of transmitting the first audio data from the first sensing system or sending a first frequency decrease request to the second sensing system to decrease a frequency of transmitting the second audio data from the second sensing system, on a condition that the second audio data is preferred based on the user preference settings, and, on a condition that the second audio data is preferred based on the user preference settings, send a second frequency increase request to the second sensing system to increase a frequency of transmitting the second audio data from the second sensing system or sending a second frequency decrease request to the first sensing system to decrease a frequency of transmitting the first audio data from the first sensing system. Advantageously, the system can handle multiple audio data sources according to user preferences.

[0016] The adjustment instructions may include surgical step instructions indicating the surgical step, and the processor may be configured to receive first audio data associated with a first healthcare professional (HCP) role in the operating room, receive second audio data associated with a second HCP role in the operating room, determine whether the first audio data associated with the first HCP is related to the surgical step indicated in the surgical step instructions and whether the second audio data associated with the first HCP is related to the surgical step indicated in the surgical step instructions, and adjust the AR content by passing the first audio data associated with the first HCP and blocking the second audio data from the second HCP if it is determined that the first audio data associated with the first HCP is related to the surgical step and the second audio data associated with the first HCP is not related to the surgical step indicated in the surgical step instructions. Advantageously, the system can handle multiple audio data sources according to the surgical steps.

[0017] The computer-implemented method may include receiving audio data from a sensing system in an operating room, the audio data including measurement data; generating augmented reality (AR) content based on the received audio data; obtaining or receiving adjustment instructions indicating adjustment information for the AR content; and adjusting the generated AR content based on the adjustment instructions.

[0018] The audio data may include measurement data that may be associated with a user, such as a medical professional, HCP (e.g., surgeon), or patient.

[0019] Advantageously, audible information associated with particular measurement data relevant and / or important to the current surgical procedure may be adjusted or amplified (e.g., increased in volume), which may accordingly improve perception of the surgical procedure, the atmosphere in the OR, and / or interaction between HCPs.

[0020] The adjustment instructions may be received from the surgical computing system and may include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a voice source location instruction indicating the voice source location of the voice data.

[0021] The adjustment instructions may include a significance of the surgical step, and the method may include identifying an audio AR setting associated with the significance of the surgical step and adjusting the AR content according to the audio AR setting. Advantageously, the AR content may be adjusted according to the significance of the surgical step, which may improve perception of the surgical procedure.

[0022] The adjustment instructions may optionally include audio information for critical surgical steps, and to adjust the generated AR content, the method may include muting audio data from the sensing system and, optionally, amplifying audio associated with the audio information for the critical surgical steps. Advantageously, the audibility of audio information that may be related to critical surgical steps may be increased, improving awareness of the surgical procedure.

[0023] The audio information for the critical surgical step may further include at least one of an increase frequency instruction or a decrease frequency instruction, and the method may include sending an increase frequency request to the sensing system based on the audio information including the increase frequency instruction, wherein the increase frequency request includes a request to increase the frequency of transmitting audio data from the sensing system, or sending a decrease frequency request to the sensing system based on the audio information including the decrease frequency instruction, wherein the decrease frequency request includes a request to decrease the frequency of transmitting audio data from the sensing system. Advantageously, audibility of audio information that may be associated with critical surgical steps may be increased, improving awareness of the surgical procedure.

[0024] The method may include receiving ambient noise in an operating room and blocking the received ambient noise. Advantageously, audio data may be received that may include ambient noise in the OR (e.g., HCPs talking to each other, sounds of surgical instruments, etc.) that may be canceled and / or blocked.

[0025] The adjustment instructions may include surgical step instructions indicating the surgical steps, and the method may include receiving first audio data associated with a first healthcare professional (HCP) role in the operating room, receiving second audio data associated with a second HCP role in the operating room, and adjusting the AR content by allowing the first audio data associated with the first HCP and blocking the second audio data from the second HCP based on the surgical step instructions. Advantageously, the system can handle multiple audio data sources according to the surgical steps.

[0026] The computer-readable medium may contain instructions that, when executed by a computer, cause the computer to perform the above-described methods.

[0027] An audible augmented reality (AR) computing system may receive audio data, such as audible information, from a sensing system in an operating room (OR). In examples, the audio data may or may not include measurement data associated with a user. The user may be or may include a healthcare professional (HCP), such as a surgeon, or a patient. In examples, the audio data may be or may include ambient noise in the OR. In examples, the audio data may block and / or cancel ambient noise in the OR. In examples, the audio data may be or may include music, such as calming music, audible feedback information, audible information associated with surgical steps and / or tasks, and / or other audible information associated with the surgical procedure.

[0028] The audio AR computing system may generate AR content. For example, the audio AR computing system may generate AR content based on audible data. The AR content may be or include audible information associated with the audio data. The AR content may be or include audible AR information transmitted to the audio AR computing system and / or a user wearing a headset controlled by the audio AR computing system. For example, the audible AR computing system may be or include a surgeon sensing system and / or a patient sensing system.

[0029] The audio AR computing system may obtain adjustment instructions. The adjustment instructions may be or include adjustment information for the AR content. For example, the adjustment instructions may be or include one or more of a surgical task instruction, a task importance instruction, a sound insertion instruction, a sound conversion instruction, and a sound source location instruction. The surgical task instruction may indicate a surgical task being performed or to be performed. The task importance instruction may indicate the importance of the surgical task. The sound insertion instruction may indicate a calming sound and / or music insertion. The sound conversion instruction may indicate sound conversion of the audio data. The sound source location instruction may indicate the sound source location of the audio data. For example, the adjustment instructions may be obtained (e.g., received) from a surgical computing system, such as a surgical hub.

[0030] The audio AR computing system may adjust the generated AR content based on, for example, the adjustment instructions. In an example, the audio AR computing system may adjust the generated AR content based on the adjustment instructions indicating the importance of a surgical step. The audio AR computing system may identify an audio AR setting associated with the importance of the surgical step. The audio AR setting may include a volume associated with audible information (e.g., associated with the surgical step), a frequency of transmission of the audible information, and / or a voice associated with the audible information. The audio AR computing system may adjust the generated AR content associated with the audio AR setting. For example, the audio AR computing system may adjust the volume of the AR content based on the AR setting. The audio AR computing system may increase or decrease the volume of the AR content based on the AR setting. The audio AR computing system may increase or decrease the frequency of receiving the generated content. The audio AR computing system may change the voice of the generated AR content based on the AR setting.

[0031] In an example, the audio AR computing system may adjust the generated AR content based on the adjustment instruction indicating audio information for the critical surgical step. The audio AR computing system may mute audio data from the sensing system. For example, the audio AR computing system may adjust the AR content by blocking (e.g., temporarily blocking) audio data from the sensing system and allowing audible information associated with the critical surgical step. The audio AR computing system may increase the volume of the audio information for the critical surgical step. A user of the audio AR computing system may receive the audible information for the critical surgical step and may listen to and / or focus on the audible information associated with the critical surgical step.

[0032] In an example, the audio AR computing system may adjust the generated AR content based on the audio information for the critical surgical step. The audio information for the critical surgical step may be or may include an increase frequency instruction or a decrease frequency instruction (e.g., AR settings, etc.). The audio AR computing system may increase the frequency of the audible information for the critical surgical step based on the increase frequency instruction. The audio AR computing system may send an increase frequency request to other computing systems (e.g., the surgical computing system and / or the central computing system) requesting them to increase the frequency of transmitting the audible information. The audio AR computing system may decrease the frequency of the audible information for the critical surgical step based on the decrease frequency instruction. The audio AR computing system may send a decrease frequency request to other computing systems (e.g., the surgical computing system, the central computing system, and / or the surgical hub) requesting them to decrease the frequency of transmitting the audible information for the critical surgical step.

[0033] In an example, the audio AR computing system may adjust the generated AR content based on surgical task instructions included (e.g., contained) in the adjustment instructions. The surgical task instructions may indicate a surgical task being performed or to be performed. The audio AR computing system may identify an audio AR setting associated with other audible information associated with the surgical task. The audio AR computing system may adjust the AR content associated with the identified audio AR setting. For example, as described herein, the audio AR computing system may adjust the AR content by increasing or decreasing the volume of the audible information associated with the surgical task and / or increasing or decreasing the frequency of receiving the audible information.

[0034] In an example, the audio AR computing system may adjust the generated AR content based on surgical task instructions included (e.g., contained) in the adjustment instructions. The surgical task instructions may indicate a surgical task being performed or to be performed. The audio AR computing system may identify the relevance of audio data from a sensing system to the surgical task indicated in the surgical task instructions. For example, the audio AR computing system may receive one or more measurement data from one or more sensing systems in an OR. The audio AR computing system may identify (e.g., determine) the relevance of audio data associated with the measurement data from the sensing systems. The audio AR computing system may determine whether to block audio data from the sensing systems. For example, the audio AR computing system may determine whether to block audio data from one or more sensing systems based on the identified relevance to the surgical task indicated in the surgical task instructions. If the audio AR computing system determines that one or more pieces of audio data from one or more sensing systems are unrelated to the surgical task indicated in the surgical task instructions, the audio AR computing system may block the one or more pieces of audio data. If the audio AR computing system determines that one or more pieces of audio data from one or more sensing systems are related to the surgical task indicated in the surgical task instructions, the audio AR computing system may authorize the one or more pieces of audio data and play the audio data.

[0035] In an example, the audio AR computing system may adjust the generated AR content by blocking ambient noise in the OR. For example, the audio AR computing system may receive audio data that may include ambient noise in the OR (e.g., HCPs talking to each other, sounds of surgical instruments, etc.). The audio AR computing system may cancel and / or block the ambient noise.

[0036] In an example, an audio AR computing system may receive audible data that has ambient noise blocked out, and generate AR content without the ambient noise.

[0037] In an example, the audio AR computing system may receive an ambient noise level indication. The ambient noise level indication may indicate the ambient noise level of the OR. In an example, the audio AR computing system may detect the ambient noise level of the OR. If the audio AR computing system determines that the ambient noise level of the OR is below a threshold ambient noise level, the audio AR computing system may recognize and / or determine that a critical step and / or task should be performed. The audio AR computing system may send a critical task indication to another computing system (e.g., a surgical computing system). The critical task indication may indicate that a critical surgical task should be performed. The audio AR computing system may adjust the AR content and receive audible information for the critical surgical task.

[0038] In an example, the audio AR computing system may request user input to adjust the AR content. For example, the audio AR computing system may request user input before adjusting the AR content. The audio AR computing system may send the user input request to another computing system (e.g., a surgical computing system). The user input request may request user input before adjusting the generated AR content. The audio AR computing system may wait a preconfigured time for a response to the user input. If the audio AR computing system does not receive a response to the user input, the audio AR computing system may send a reminder user input request and / or send the user input request to another HCP in the OR.

[0039] In an example, the audio AR computing system may receive one or more audio data. For example, the audio AR computing system may receive audible information from a sensing system and other audible information from another sensing system. The audio AR computing system may obtain adjustment instructions. The adjustment instructions may be or may include user preference settings associated with a surgical procedure. Based on the user preference settings, the audio AR computing system may adjust the AR content by selecting and / or receiving audio information from a sensing system (e.g., first audible information from a first sensing system). The audio AR computing system may block other audio information from another sensing system (e.g., second audible information from a second sensing system).

[0040] The user preference setting may indicate preferred audio data when the AR computing system receives audible data from multiple sensing systems. The AR computing system may adjust the AR content by increasing the volume of selected and / or preferred audio data. The audio AR computing system may decrease the volume of unselected audio data. The audio AR computing system may cancel and / or block unselected audio data. The audio AR computing system may request an increase in the frequency of receiving selected and / or preferred audio data. The audio AR computing system may request a decrease in the frequency of receiving unselected audio data.

[0041] In an example, the audio AR computing system may adjust the AR content based on adjustment instructions indicating surgical step instructions. The surgical step instructions may indicate a current surgical step associated with the surgical procedure. The audio AR computing system may receive audio data associated with an HCP role in the OR. The audio AR computing system may receive other audio data associated with other HCP roles in the OR. The audio AR computing system may adjust the AR content to allow audio data (e.g., first audio data) associated with the HCP role (e.g., first HCP role) and block other audio data (e.g., second audio data) associated with the other HCP role (e.g., second HCP role). [Brief explanation of the drawings]

[0042] [Figure 1A] FIG. 1 is a block diagram of a computer-implemented patient and surgeon monitoring system. [Figure 1B] FIG. 1 is a block diagram of an exemplary relationship between a sensing system, a biomarker, and a physiological system. [Figure 2A] 1 illustrates an example of a surgeon monitoring system in a surgical operating room. [Figure 2B] 1 illustrates an example of a patient monitoring system (eg, a control patient monitoring system). [Figure 2C] 1 illustrates an example of a patient monitoring system (eg, a non-controlled patient monitoring system). [Figure 3] 1 illustrates an exemplary surgical hub paired with various systems. [Figure 4] 1 illustrates a surgical data network having a set of communicating surgical hubs configured to connect with a set of sensing systems, an environmental sensing system, a set of devices, and the like. [Figure 5] 1 illustrates an exemplary computer-implemented interactive surgical system that may be part of a surgeon monitoring system. [Figure 6A] 1 illustrates a surgical hub with multiple modules coupled to a modular control tower. [Figure 6B] 1 illustrates an example of a controlled patient monitoring system. [Figure 6C] 1 illustrates an example of an uncontrolled patient monitoring system. [Figure 7A] 1 shows a logic diagram of a control system for a surgical instrument or tool. [Figure 7B] 1 illustrates an exemplary sensing system having a sensor unit and a data processing and communication unit. [Figure 7C] 1 illustrates an exemplary sensing system having a sensor unit and a data processing and communication unit. [Figure 7D] 1 illustrates an exemplary sensing system having a sensor unit and a data processing and communication unit. [Figure 8] 10 illustrates an exemplary timeline of an exemplary surgical procedure illustrating adjusting operating parameters of a surgical device based on surgeon biomarker levels. [Figure 9] FIG. 1 is a block diagram of a computer-implemented interactive surgeon / patient monitoring system. [Figure 10] 1 illustrates an exemplary surgical system including a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter. [Figure 11A] 1 illustrates an example of a sensing system that may be used to monitor surgeon biomarkers or patient biomarkers. [Figure 11B] 1 illustrates an example of a sensing system that may be used to monitor surgeon biomarkers or patient biomarkers. [Figure 11C] 1 illustrates an example of a sensing system that may be used to monitor surgeon biomarkers or patient biomarkers. [Figure 11D] 1 illustrates an example of a sensing system that may be used to monitor surgeon biomarkers or patient biomarkers. [Figure 12] FIG. 1 is a block diagram of a patient or surgeon monitoring system. [Figure 13] 1 illustrates an example flow of an audio augmented reality (AR) computing system for adjusting AR content. DETAILED DESCRIPTION OF THE INVENTION

[0043] The applicant of the present application owns the following concurrently filed US patent applications, each of which is incorporated herein by reference in its entirety: U.S. Patent Application No. 16 / 209,416, filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS"; U.S. Patent Application No. 15 / 940,671, entitled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER," filed March 29, 2018 (Attorney Docket No. END8502USNP); U.S. Patent Application No. 16 / 182,269, entitled "IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE," filed November 6, 2018 (Attorney Docket No. END9018USNP3); U.S. Patent Application No. 16 / 729,747, entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS," filed December 31, 2019 (Attorney Docket No. END9217USNP1); U.S. Patent Application No. 16 / 729,778, entitled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE," filed December 31, 2019 (Attorney Docket No. END9219USNP1); U.S. Patent Application No. 16 / 729,807, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY," filed December 31, 2019 (Attorney Docket No. END9228USNP1); U.S. Patent Application No. 15 / 940,654, entitled "SURGICAL HUB SITUATIONAL AWARENESS," filed March 29, 2018 (Attorney Docket No. END8501USNP); U.S. Patent Application No. 15 / 940,671, entitled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER," filed March 29, 2018 (Attorney Docket No. END8502USNP); U.S. Patent Application No. 15 / 940,704 (Attorney Docket No. END8504USNP), entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT," filed March 29, 2018; U.S. Patent Application No. 16 / 182,290, entitled "SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION," filed November 6, 2018 (Attorney Docket No. END9018USNP5); U.S. Patent No. 9,011,427, entitled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued April 21, 2015; U.S. Patent No. 9,123,155, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES," issued September 1, 2015; U.S. Patent Application No. 16 / 209,478, filed December 4, 2018, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE" (Attorney Docket No. END9015USNP1); and U.S. Patent Application No. 16 / 182,246, entitled "ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES," filed November 6, 2018 (Attorney Docket No. END9016USNP1).

[0044] FIG. 1A is a block diagram of a computer-implemented patient and surgeon monitoring system 20000. The patient and surgeon monitoring system 20000 may include one or more surgeon monitoring systems 20002 and one or more patient monitoring systems (e.g., one or more control patient monitoring systems 20003 and one or more non-control patient monitoring systems 20004). Each surgeon monitoring system 20002 may include a computer-implemented interactive surgical system. Each surgeon monitoring system 20002 may include at least one of the following: a surgical hub 20006 in communication with a cloud computing system 20008, for example, as described in FIG. 2A. Each of the patient monitoring systems may include at least one of the following: a surgical hub 20006 or a computing device 20016 in communication with a cloud computing system 20008, for example, as further described in FIGS. 2B and 2C. The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Each of the surgeon monitoring system 20002, the controlling patient monitoring system 20003, or the non-controlling patient monitoring system 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more surgeon sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include, for example, one or more devices used to measure one or more environmental attributes, e.g., as further described in FIG. 2A . The robotic system 20013 (same as 20034 in FIG. 2A ) may include multiple devices used to perform a surgical procedure, e.g., as further described in FIG. 2A .

[0045] The surgical hub 20006 may have cooperative interaction with one or more means for displaying images from the laparoscopic scope and information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 may interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 may be configured to collect measurement data from one or more sensing systems 20011 and send notification or control messages to one or more sensing systems 20011. The surgical hub 20006 may send and / or receive information, including notification information, to a human interface system 20012. The human interface system 20012 may include one or more human interface devices (HIDs). The surgical hub 20006 may send and / or receive audio, display, and / or control information to various devices in communication with the surgical hub.

[0046] 1B is a block diagram of an example relationship between a sensing system 20001, biomarkers 20005, and a physiological system 20007. This relationship may be employed in a computer-implemented patient and surgeon monitoring system 20000 and the systems, devices, and methods disclosed herein. For example, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more surgeon sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, as discussed in FIG. 1A. The one or more sensing systems 20001 may measure data related to various biomarkers 20005. The one or more sensing systems 20001 may measure the biomarkers 20005 using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The one or more sensors may measure the biomarkers 20005 described herein using one or more of the following sensing technologies: photoplethysmography, electrocardiography, electroencephalography, colorimetry, obstruction testing, potentiometry, amperometry, etc.

[0047] The biomarkers 20005 measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen uptake, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood hydrogen ion concentration, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, coughing and sneezing, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental status, sweat, circulating tumor cells, autonomic tone, circadian rhythm, and / or menstrual cycle.

[0048] The biomarkers 20005 may relate to physiological systems 20007, which may include, but are not limited to, behavioral and psychological, cardiovascular, urinary, dermatological, nervous, digestive, respiratory, endocrine, immune, oncological, skeletal, and / or reproductive systems. Information from the biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgeon monitoring system 20000. Information from the biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgeon monitoring system 20000 to improve the system and / or to improve patient outcomes.

[0049] FIG. 2A illustrates an example of a surgeon monitoring system 20002 in a surgical operating room. As shown in FIG. 2A, a patient is being operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more surgeon sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding the HCPs may also be monitored by one or more environmental sensing systems including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors, which may be deployed in the operating room. The surgeon sensing system 20020 and the environmental sensing systems may communicate with a surgical hub 20006, which may communicate with one or more cloud servers 20009 of a cloud computing system 20008, as shown in FIG. 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as the HCP's position within the operating room, the HCP's movement, ambient noise within the operating room, and temperature / humidity within the operating room.

[0050] As shown in FIG. 2A , a primary display 20023 and one or more audio output devices (e.g., speaker 20019) are positioned in the sterile field for visibility to the operator of the operating table 20024. Additionally, a visualization / notification tower 20026 is positioned outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human interactive device (HID) 20027 and a second non-sterile HID 20029, which may face opposite each other. The HIDs may be displays or displays with touchscreens that allow humans to directly interact with the HIDs. A human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to coordinate the flow of information to operators inside and outside the sterile field. In one example, the surgical hub 20006 may cause an HID (e.g., primary HID 20023) to display notifications and / or information regarding the patient and / or surgical procedure steps. In one example, the surgical hub 20006 may prompt and / or receive input from personnel within the sterile field or non-sterile area. In one example, the surgical hub 20006 may cause the HIDs to display snapshots of the surgical site captured by the imaging device 20030 on the non-sterile HID 20027 or 20029 while maintaining a live video of the surgical site on the main HID 20023. The snapshots on the non-sterile displays 20027 or 20029 may, for example, enable the non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0051] In one aspect, the surgical hub 20006 may be configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to the primary display 20023 in the sterile field for viewing by the sterile operator at the operating table. In one example, the input may be in the form of a modification to a snapshot displayed on the non-sterile display 20027 or 20029, which may be sent by the surgical hub 20006 to the primary display 20023.

[0052] 2A , a surgical instrument 20031 is used as part of a surgeon monitoring system 20002 in a surgical procedure. A hub 20006 may be configured to coordinate information flow to the display of the surgical instrument 20031. See, for example, U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 can be sent by the hub 20006 to the surgical instrument display in the sterile field, where it can be viewed by the operator of the surgical instrument 20031. Exemplary surgical instruments suitable for use with surgical system 20002 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0053] FIG. 2A shows an example of a surgical system 20002 being used to perform a surgical procedure on a patient lying on an operating table 20024 in a surgical operating room 20035. A robotic system 20034 may be used as part of the surgical system 20002 in the surgical procedure. The robotic system 20034 may include a surgeon's console 20036, a patient side cart 20032 (surgical robot), and a surgical robot hub 20033. The patient side cart 20032 can manipulate at least one detachably coupled surgical tool 20037 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 20036. Images of the surgical site can be acquired by a medical imaging device 20030, which can be manipulated by the patient side cart 20032 to orient the imaging device 20030. The robotic hub 20033 can be used to process images of the surgical site for subsequent display to the surgeon through the surgeon's console 20036.

[0054] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0055] Various examples of cloud-based analytics methods implemented by cloud computing system 20008 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0056] In various embodiments, the imaging device 20030 may include at least one image sensor and at least one optical component. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0057] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0058] The one or more illumination sources may be configured to emit electromagnetic energy within the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and is sometimes referred to as visible light or simply light. The typical human eye responds to wavelengths in the air ranging from about 380 nm to about 750 nm.

[0059] The invisible spectrum (e.g., non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0060] In various aspects, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0061] The imaging device may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging can enable the extraction of additional information that the human eye cannot capture with its red, green, and blue receptors. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is complete to perform one or more of the above-mentioned tests on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical equipment is necessary in any surgical procedure. The strict hygiene and sterilization conditions required in the "surgical field," i.e., the operating room or procedure room, require the highest possible sterility of all medical devices and equipment. Part of the above sterilization process includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 20030 and its accessories and components. It is understood that the sterile field can be considered a specific area deemed free of microorganisms, such as in a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding the patient prepared for the surgical procedure. The sterile field can include cleaned team members wearing appropriate clothing, as well as all equipment and fixtures within the area.

[0062] The wearable sensing system 20011 shown in FIG. 1 may include one or more sensing systems, such as a surgeon sensing system 20020 as shown in FIG. 2A. The surgeon sensing system 20020 may include a sensing system for monitoring and detecting a set of physical conditions and / or a set of physiological conditions of a healthcare provider (HCP). An HCP may generally be a surgeon or one or more medical personnel assisting the surgeon or other healthcare provider. In one example, the sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In another example, the sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movement and / or shaking and determine the magnitude and frequency of tremors. The sensing system 20020 may transmit measurement data associated with the set of biomarkers and data associated with the surgeon's physical condition to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing devices may include a camera 20021 for detecting the HCP's hand / body position. The environmental sensing devices may include a microphone 20022 for measuring ambient noise within the surgical field. Other environmental sensing devices may include devices such as a thermometer for measuring temperature, a hygrometer for measuring ambient humidity within the surgical field, etc. Either alone or in communication with a cloud computing system, the surgical hub 20006 may use surgeon biomarker measurement data and / or environmental sensing information to modify the average latency of a handheld instrument's control algorithm or robotic interface, for example, to minimize tremor. In one example, the surgeon sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit measurement data associated with the surgeon biomarkers to the surgical hub 20006.The surgeon sensing system 20020 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Wi-Fi. Surgeon biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the surgical field may include ambient noise levels associated with the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention level, etc.

[0063] The surgical hub 20006 may use surgeon biomarker measurement data associated with the HCP to adaptively control one or more surgical instruments 20031. For example, the surgical hub 20006 may send a control program to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may send the control program based on situational awareness and / or context regarding the importance or criticality of the task. The control program may instruct the instrument to modify its operation to provide more control when control is needed.

[0064] FIG. 2B illustrates an example of a patient monitoring system 20003 (e.g., a control patient monitoring system). As shown in FIG. 2B, a patient in a controlled environment (e.g., a hospital recovery room) may be monitored by multiple sensing systems (e.g., patient sensing system 20041). Patient sensing system 20041 (e.g., a headband) may be used to measure electroencephalograms (EEG) to measure the patient's brain's electrical activity. Patient sensing system 20042 may be used to measure various biomarkers of the patient, including, for example, heart rate, VO2 level, etc. Patient sensing system 20043 (e.g., a flexible patch attached to the patient's skin) may be used to measure sweat lactate and / or potassium levels by analyzing small amounts of sweat captured from the skin's surface using microfluidic channels. Patient sensing system 20044 (e.g., a wristband or watch) may be used to measure blood pressure, heart rate, heart rate variability, VO2 level, etc. using various technologies, as described herein. The patient sensing system 20045 (e.g., a ring) may be used to measure peripheral temperature, heart rate, heart rate variability, VO2 level, etc. using various technologies, as described herein. The patient sensing systems 20041-20045 may use a radio frequency (RF) link to communicate with the surgical hub 20006. The patient sensing systems 20041-20045 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Thread, Wi-Fi, etc.

[0065] The sensing systems 20041-20045 can communicate with a surgical hub 20006, which can communicate with a remote server 20009 of a remote cloud computing system 20008. The surgical hub 20006 also communicates with an HID 20046. The HID 20046 can display measurement data associated with one or more patient biomarkers. For example, the HID 20046 can display blood pressure, oxygen saturation levels, respiratory rate, etc. The HID 20046 can also display notifications for the patient or HCP providing information about the patient, such as information about recovery milestones or complications. In one example, the information about recovery milestones or complications can be associated with a surgical procedure the patient may have undergone. In one example, the HID 20046 can display instructions for the patient to perform an activity. For example, the HID 20046 can display inhalation and exhalation instructions. In one example, the HID 20046 can be part of the sensing system.

[0066] 2B, the patient and the environment surrounding the patient may be monitored by one or more environmental sensing systems 20015 including, for example, a microphone (e.g., to detect ambient noise associated with or around the patient), temperature / humidity sensors, a camera to detect the patient's breathing patterns, etc. The environmental sensing system 20015 may communicate with a surgical hub 20006, which in turn communicates with a remote server 20009 of a remote cloud computing system 20008.

[0067] In one example, the patient sensing system 20044 may receive notification information from the surgical hub 20006 for display on the display unit or HID of the patient sensing system 20044. The notification information may include, for example, in the case of post-operative recovery, notifications regarding recovery milestones or notifications regarding complications. In one example, the notification information may include an actionable severity level associated with the notification. The patient sensing system 20044 may display the notification and the actionable severity level to the patient. The patient sensing system may alert the patient using haptic feedback. The visual and / or haptic notification may be accompanied by an audible notification prompting the patient to pay attention to the visual notification provided on the display unit of the sensing system.

[0068] FIG. 2C illustrates an example of a patient monitoring system (e.g., uncontrolled patient monitoring system 20004). As shown in FIG. 2C, a patient in an uncontrolled environment (e.g., the patient's residence) is monitored by multiple patient sensing systems 20041-20045. Patient sensing systems 20041-20045 may measure and / or report measurement data associated with one or more patient biomarkers. For example, patient sensing system 20041, a headband, may be used to measure electroencephalograms (EEG). Other patient sensing systems 20042, 20043, 20044, and 20045 are examples where various patient biomarkers are reported, measured, and / or reported, as shown in FIG. 2B. One or more of patient sensing systems 20041-20045 may transmit measurement data associated with the monitored patient biomarkers to computing device 20047, which can communicate with remote server 20009 of remote cloud computing system 20008. The patient sensing systems 20041-20045 may use a radio frequency (RF) link to communicate with the computing device 20047 (e.g., a smartphone, a tablet, etc.). The patient sensing systems 20041-20045 may use one or more of the following RF protocols to communicate with the computing device 20047: Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Thread, Wi-Fi, etc. In one example, the patient sensing systems 20041-20045 may be connected to the computing device 20047 via a wireless router, a wireless hub, or a wireless bridge.

[0069] The computing device 20047 can communicate with a remote server 20009 that is part of a cloud computing system 20008. In one example, the computing device 20047 may communicate with the remote server 20009 through an internet service provider's cable / FIOS networking node. In one example, the patient sensing system may communicate directly with the remote server 20009. The computing device 20047 or sensing system may communicate with the remote server 20009 through a cellular transmission / reception point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR) or 5G.

[0070] In one example, the computing device 20047 may display information associated with patient biomarkers. For example, the computing device 20047 may display blood pressure, oxygen saturation level, respiratory rate, etc. The computing device 20047 may display notifications for the patient or HCP providing information about the patient, such as information about recovery milestones or complications.

[0071] In one example, the computing device 20047 and / or the patient sensing system 20044 may receive notification information from the surgical hub 20006 for display on a display unit of the computing device 20047 and / or the patient sensing system 20044. The notification information may include, for example, in the case of post-operative recovery, notifications regarding recovery milestones or notifications regarding complications. The notification information may also include an actionable severity level associated with the notification. The computing device 20047 and / or the sensing system 20044 may display the notification and the actionable severity level to the patient. The patient sensing system may also alert the patient using haptic feedback. The visual and / or haptic notification may be accompanied by an audible notification prompting the patient to pay attention to the visual notification provided on the display unit of the sensing system.

[0072] 3 illustrates an exemplary surgeon monitoring system 20002 having a surgical hub 20006 paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communications module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain embodiments, as shown in FIG. 3, the hub 20006 further includes a smoke evacuation module 20054 and / or a suction / irrigation module 20055. During a surgical procedure, the application of energy to tissue for sealing and / or cutting is commonly associated with smoke evacuation, aspiration of excess fluid, and / or irrigation of tissue. Fluid, power, and / or data lines from different sources often become tangled during surgical procedures. Addressing this issue can result in valuable time being lost during a surgical procedure. Untangling the lines may require unplugging them from their corresponding modules, which may require resetting the modules. The hub modular enclosure 20060 provides a unified environment for managing power, data, and fluid lines, reducing the frequency of such tangles between the lines. An embodiment of the present disclosure presents a surgical hub 20006 for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combination generator module slidably receivable within a docking station of the hub enclosure 20060. The docking station includes data and power contacts. The combination generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit.In one aspect, the combination generator module also includes a smoke evacuation component, at least one energy delivery cable for connecting the combination generator module to a surgical instrument, at least one smoke evacuation component configured to evacuate smoke, fluid, and / or particulates generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the smoke evacuation component. In one aspect, the fluid line can be a first fluid line, and a second fluid line extends from the remote surgical site to the aspiration and irrigation module 20055 slidably received within the hub enclosure 20060. In one aspect, the hub enclosure 20060 can include a fluid interface. Some surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another, different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut the sealed tissue. Aspects of the present disclosure present a solution in which a hub modular enclosure 20060 is configured to house various generators and facilitate bidirectional communication therebetween. One advantage of the hub modular enclosure 20060 is that it allows for rapid removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in a surgical procedure involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station with a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable into electrical engagement with the power and data contacts and the first energy generator module is slidably movable out of electrical engagement with the first power and data contacts.Further to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power contacts and the data contacts, and the second energy generator module is slidably movable out of electrical engagement with the second power contacts and the second data contacts. In addition, the modular surgical enclosure also includes a communication bus between the first and second docking ports configured to facilitate communication between the first and second energy generator modules. With reference to FIG. 3 , an embodiment of the present disclosure is presented relating to a hub modular enclosure 20060 that enables modular integration of a generator module 20050, a smoke evacuation module 20054, and a suction / irrigation module 20055. The hub modular enclosure 20060 further facilitates intercommunication between the modules 20059, 20054, and 20055. The generator module 20050 may be a generator module 20050 comprising integrated monopolar, bipolar, and ultrasonic components supported within a single housing unit slidably insertable into the hub modular enclosure 20060. The generator module 20050 may be configured to connect to a monopolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may comprise a series of monopolar, bipolar, and / or ultrasonic generator modules that interact via the hub modular enclosure 20060. The hub modular enclosure 20060 may be configured to facilitate the insertion of multiple generators and interactive communication between the generators docked to the hub modular enclosure 20060 such that the multiple generators function as a single generator.

[0073] FIG. 4 illustrates a surgical data network having a set of communication hubs configured to connect a set of sensing systems, environmental sensing systems, and other modular devices located in one or more surgical sites, patient recovery rooms, or rooms within a medical facility equipped for surgical procedures to a cloud, in accordance with at least one embodiment of the present disclosure.

[0074] As shown in FIG. 4 , the surgical hub system 20060 may include a modular communications hub 20065 configured to connect modular devices located at the medical facility to a cloud-based system (e.g., a cloud computing system 20064, which may include a remote server 20067 coupled to remote storage 20068). The modular communications hub 20065 and devices may be connected in a room within the medical facility specially equipped for surgical procedures. In one aspect, the modular communications hub 20065 may include a network hub 20061 and / or a network switch 20062 in communication with a network router 20066. The modular communications hub 20065 may be coupled to a local computer system 20063 to provide local computer processing and data manipulation. The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, allowing data to go from one device (or segment) to another and to cloud computing resources. The intelligent surgical data network includes additional mechanisms that allow traffic to pass through the monitored surgical data network and configure each port in the network hub 20061 or network switch 20062. The intelligent surgical data network may be referred to as a manageable hub or switch. The switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0075] Modular devices 1a-1n located at the surgical site may be coupled to a modular communication hub 20065. The network hub 20061 and / or network switch 20062 may be coupled to a network router 20066 to connect the devices 1a-1n to a cloud computing system 20064 or a local computer system 20063. Data associated with the devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with the devices 1a-1n may also be transferred to the local computer system 20063 for local data processing and manipulation. Modular devices 2a-2m located at the same surgical site may also be coupled to the network switch 20062. The network switch 20062 may be coupled to the network hub 20061 and / or network router 20066 to connect the devices 2a-2m to the cloud 20064. Data associated with the devices 2a-2m may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. Data associated with the devices 2a-2m may also be transferred to a local computer system 20063 for local data processing and manipulation.

[0076] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing systems 20069 may include a surgeon sensing system and / or a patient sensing system. The one or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly through one of the network routers 20066 or through a network hub 20061 or network switching 20062 that communicates with the network router 20066.

[0077] The sensing system 20069 may be coupled to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 may be transferred via the network router 20066 to the cloud computing system 20064 for data processing and manipulation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0078] 4, the surgical hub system 20060 may be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. The modular communications hub 20065 may be housed within a modular control tower configured to receive multiple devices 1a-1n / 2a-2m. A local computer system 20063 may also be housed within the modular control tower. The modular communications hub 20065 may be connected to a display 20068 for displaying images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include various modules such as, for example, an imaging module coupled to an endoscope, a generator module coupled to an energy-based surgical device, a smoke evacuation module, a suction / irrigation module, a communication module, a processor module, a storage array, a surgical device coupled to a display, and / or a non-contact sensor module, among other modular devices that may be connected to the modular communication hub 20065 of the surgical data network.

[0079] In one aspect, the surgical hub system 20060 shown in FIG. 4 may comprise a combination of network hub(s), network switch(es), and network router(s) that connect the devices 1a-1n / 2a-2m or sensing system 20069 to a cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or sensing system 20069 coupled to the network hub 20061 or network switch 20062 can collect data or measurement data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to run software applications. The term "cloud" may be used as a metaphor for the "Internet," but the term is not so limited. Accordingly, the term "cloud computing" may be used herein to refer to a type of internet-based computing in which various services, such as servers, storage, and applications, are delivered via the internet to a modular communications hub 20065 and / or computer system 20063 located at a surgical site (e.g., a fixed, mobile, temporary, or on-site operating room or space) and devices connected to the modular communications hub 20065 and / or computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, a cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located within one or more surgical sites. Cloud computing services may perform numerous calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located within the surgical site. Hub hardware allows multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0080] By applying cloud computing data processing techniques to data collected by devices 1a-1n / 2a-2m, a surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe tissue status and evaluate leakage or perfusion of sealed tissue after tissue sealing and cutting procedures. Using cloud-based computing, at least some of devices 1a-1n / 2a-2m can be used to diagnostically examine data including images of bodily tissue samples to identify pathologies, such as the effects of disease. Such data can include tissue localization and margin confirmation, as well as phenotyping. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to cloud computing system 20064 or local computer system 20063, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcome of the surgical procedure by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics, can be performed on tissue-specific sites and conditions. Such data analysis may further employ prognostic analysis processes, and the use of standardized techniques can provide useful feedback to either confirm or suggest modifications to surgical treatments and surgeon performance.

[0081] By applying cloud computer data processing techniques to the measurement data collected by the sensing systems 20069, the surgical data network can provide improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 may be employed to assess the physiological status of a surgeon operating on a patient, or a patient being prepared for a surgical procedure, or a patient recovering after a surgical procedure. The cloud-based computing system 20064 may be used to monitor biomarkers associated with the surgeon or patient, generate a surgical plan based on measurement data collected at least prior to the surgical procedure, provide control signals to surgical instruments during the surgical procedure, and notify the patient of complications during the post-operative period.

[0082] The surgical field devices 1a-1n may be connected to the modular communication hub 20065 via wired or wireless channels depending on the configuration of the devices 1a-1n relative to the network hub 20061. The network hub 20061 may, in one aspect, be implemented as a local network broadcast device operating on the physical layer of the Open Systems Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 20061 may collect data in the form of packets and send them to a router in half-duplex mode. The network hub 20061 cannot store any Media Access Control / Internet Protocol (MAC / IP) protocols for forwarding device data. Only one of the devices 1a-1n can send data through the network hub 20061 at a time. The network hub 20061 cannot have a routing table or intelligence regarding where to send information and broadcasts all network data across each connection to a remote server 20067 in the cloud computing system 20064. Although the network hub 20061 can detect basic network errors such as collisions, broadcasting all information to multiple ports can pose a security risk and cause bottlenecks.

[0083] The surgical field devices 2a-2m may be connected to the network switch 20062 via wired or wireless channels. The network switch 20062 functions within the data link layer of the OSI model. The network switch 20062 may be a multicast device for connecting the devices 2a-2m located within the same surgical field to a network. The network switch 20062 may transmit data in the form of frames to the network router 20066 and may function in full-duplex mode. Multiple devices 2a-2m may transmit data simultaneously through the network switch 20062. The network switch 20062 stores and uses the MAC addresses of the devices 2a-2m to forward data.

[0084] The network hub 20061 and / or the network switch 20062 may be coupled to a network router 20066 for connection to the cloud computing system 20064. The network router 20066 functions within the network layer of the OSI model. The network router 20066 creates a path for forwarding data packets received from the network hub 20061 and / or the network switch 20062 to cloud-based computer resources for further processing and manipulation of data collected by any one or all of the devices 1a-1n / 2a-2m and the wearable sensing system 20011. The network router 20066 may be employed to connect two or more different networks located in different locations, such as different surgical sites within the same medical facility or different surgical sites within different medical facilities. The network router 20066 may transmit data in the form of packets to the cloud computing system 20064 and functions in full-duplex mode. Multiple devices can transmit data simultaneously. The network router 20066 may use IP addresses to forward data.

[0085] In one example, the network hub 20061 may be implemented as a USB hub that allows multiple USB devices to be connected to a host computer. The USB hub can expand a single USB port into several tiers so that more ports are available for connecting devices to the host system computer. The network hub 20061 may include wired or wireless functionality for receiving information via wired or wireless channels. In one aspect, a wireless USB short-range, high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and 2a-2m located within the surgical field.

[0086] In an example, the surgical field devices 1a-1n / 2a-2m and / or sensing system 20069 can communicate with the modular communications hub 20065 via the Bluetooth wireless technology standard to exchange data over short distances (using short wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) from fixed and mobile devices and create a personal area network (PAN). The surgical field devices 1a-1n / 2a-2m and / or sensing system 20069 may communicate with the modular communications hub 20065 via several wireless or wired communications standards or protocols, including, but not limited to, Bluetooth, Bluetooth Low Energy, Near Field Communication (NFC), Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long Term Evolution (LTE), and any other wireless and wired protocols designated as Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as 3G, 4G, 5G, and beyond. The computing module may include multiple communications modules. For example, the first communication module may be dedicated to shorter-range wireless communications such as Wi-Fi, Bluetooth low energy, Bluetooth, and Bluetooth Smart, and the second communication module may be dedicated to longer-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, and TDMA.

[0087] The modular communications hub 20065 can serve as a central connection for one or more of the surgical field devices 1a-1n / 2a-2m and / or the sensing system 20069 and can handle data types known as frames. The frames can carry data generated by the devices 1a-1n / 2a-2m and / or the sensing system 20069. Once a frame is received by the modular communications hub 20065, it can be amplified and / or transmitted to the network router 20066, which can forward the data to the cloud computing system 20064 or the local computer system 20063 using several wireless or wired communications standards or protocols, as described herein.

[0088] The modular communications hub 20065 can be used as a standalone device or can be connected to compatible network hubs 20061 and network switches 20062 to form a larger network. The modular communications hub 20065 can generally be easy to install, configure, and maintain, making it a good choice for networking surgical field devices 1a-1n / 2a-2m.

[0089] 5 illustrates a computer-implemented interactive surgical system 20070 that may be part of the surgeon monitoring system 20002. The computer-implemented interactive surgical system 20070 is similar in many respects to the surgeon sensing system 20002. For example, the computer-implemented interactive surgical system 20070 may include one or more surgical subsystems 20072 that are similar in many respects to the surgeon monitoring system 20002. Each surgical subsystem 20072 includes at least one surgical hub 20076 that communicates with a cloud computing system 20064 that may include a remote server 20077 and remote storage 20078. In one aspect, the computer-implemented interactive surgical system 20070 may include a modular control tower 20085 connected to multiple surgical field devices, such as sensing systems (e.g., the surgeon sensing system 20002 and / or the patient sensing system 20003), intelligent surgical instruments, robots, and other computerized devices located within the surgical field. As shown in FIG. 6A, the modular control tower 20085 may include a modular communications hub 20065 coupled to a local computing system 20063.

[0090] 5 , modular control tower 20085 may be coupled to an imaging module 20088 that may be coupled to an endoscope 20087, a generator module 20090 that may be coupled to an energy device 20089, a smoke evacuator module 20091, a suction / irrigation module 20092, a communications module 20097, a processor module 20093, a storage array 20094, a smart device / instrument 20095 optionally coupled to displays 20086 and 20084, respectively, and a non-contact sensor module 20096. Modular control tower 20085 may also communicate with one or more sensing systems 20069 and environmental sensing system 20015. Sensing system 20069 may be connected to modular control tower 20085 either directly via a router or via communications module 20097. Surgical field devices may be coupled to cloud computing resources and data storage via modular control tower 20085. The robotic surgical hub 20082 may also be connected to the modular control tower 20085 and cloud computing resources. The devices / instruments 20095 or 20084, among others, and the human interface system 20080 may be coupled to the modular control tower 20085 via wired or wireless communication standards or protocols, as described herein. The human interface system 20080 may include a display subsystem and a notification subsystem. The modular control tower 20085 may be coupled to a hub display 20081 (e.g., monitor, screen) for displaying and overlaying images received from the imaging module 20088, the device / instrument display 20086, and / or other human interface systems 20080. The hub display 20081 may also display data received from devices connected to the modular control tower 20085 along with images and overlay images.

[0091] FIG. 6A illustrates a surgical hub 20076 comprising multiple modules coupled to a modular control tower 20085. As shown in FIG. 6A, the surgical hub 20076 may be connected to a generator module 20090, a smoke evacuation module 20091, a suction / irrigation module 20092, and a communications module 20097. The modular control tower 20085 may comprise a modular communications hub 20065, e.g., a network-connected device, and a computer system 20063 for providing, e.g., local wireless connectivity with sensing systems, local processing, complication monitoring, visualization, and imaging. As shown in FIG. 6A, the modular communications hub 20065 may be connected in an expanding configuration (e.g., a tiered configuration) to extend several modules (e.g., devices) and several sensing systems 20069 that may be connected to the modular communications hub 20065 and transfer data associated with the modules and / or measurement data associated with the sensing systems 20069 to the computer system 20063, cloud computing resources, or both. 6A , each of the network hubs / switches 20061 / 20062 in the modular communications hub 20065 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to the processor 20102 to provide a communication connection to cloud computing resources and the local display 20108. At least one of the network hub switches 20061 / 20062 in the modular communications hub 20065 may have at least one wireless interface to provide a communication connection between the sensing system 20069 and / or the device 20095 and the cloud computing system 20064. Communication to the cloud computing system 20064 may occur through either a wired communication channel or a wireless communication channel.

[0092] The surgical hub 20076 may employ a non-contact sensor module 20096 to measure the dimensions of the surgical field and generate a map of the surgical field using either an ultrasonic non-contact measurement device or a laser-based non-contact measurement device. In U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," which is incorporated herein by reference in its entirety, a sensor module is configured to determine the size of the operating room and adjust the distance limit for Bluetooth pairing. As described in the "Surgical Hub Spatial Awareness Within an Operating Room" section of the same application, an ultrasonic-based non-contact sensor module may scan the surgical field by transmitting bursts of ultrasound and receiving echoes as the bursts of ultrasound reflect off the exterior walls of the operating field. A laser-based non-contact sensor module may scan the surgical field by, for example, transmitting laser light pulses, receiving the laser light pulses that reflect off the exterior walls of the operating field, and comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating field and adjust the distance limit for Bluetooth pairing.

[0093] The computer system 20063 may include a processor 20102 and a network interface 20100. The processor 20102 may be coupled to a communications module 20103, storage 20104, memory 20105, non-volatile memory 20106, and an input / output (I / O) interface 20107 via a system bus. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, using any of a variety of available bus architectures, including, but not limited to, a 9-bit bus, Industry Standard Architecture (ISA), Micro Channel Architecture (MSA), Enhanced ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer System Interface (SCSI), or any other proprietary bus.

[0094] The processor 20102 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In one embodiment, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes 256 KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle serial random access memory (SRAM), internal read-only memory (ROM) loaded with StellarisWare® software, 2 KB of electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which are available in the product datasheet.

[0095] In one example, the processor 20102 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller may be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0096] System memory can include both volatile and nonvolatile memory. The basic input / output system (BIOS), containing the basic routines for transferring information between elements within a computer system, such as during start-up, is stored in nonvolatile memory. For example, nonvolatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random access memory (RAM), which acts as external cache memory. RAM is available in many forms, including static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync link DRAM (SLDRAM), and direct RAM (DRRAM).

[0097] The computer system 20063 may also include removable / non-removable, volatile / non-volatile computer storage media, such as disk storage. Disk storage may include, but is not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage may include the above storage media, either independently or in combination with other storage media. Other storage media include, but are not limited to, optical disk drives such as compact disc ROM drives (CD-ROMs), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

[0098] It should be understood that the computer system 20063 may include software that acts as an intermediary between users and the basic computer resources described in a suitable operating environment. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate resources of the computer system. System applications may take advantage of resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0099] A user may input commands or information into the computer system 20063 through input device(s) coupled to the I / O interface 20107. Input devices may include, but are not limited to, pointing devices such as a mouse, trackball, stylus, or touchpad; keyboards; microphones; joysticks; gamepads; satellite dishes; scanners; TV tuner cards; digital cameras; digital video cameras; webcams; and the like. These and other input devices connect to the processor 20102 through the system bus via interface port(s). Interface port(s) include, for example, serial ports, parallel ports, game ports, and USBs. The output device(s) use some of the same types of ports as the input device(s). Thus, for example, a USB port may be used to provide input to the computer system 20063 and to output information from the computer system 20063 to an output device. An output adapter may be provided to illustrate that some output devices may be present, such as monitors, displays, speakers, and printers, among other output devices that may require special adapters. Output adapters may include, by way of example and not limitation, video and sound cards that provide a means of connection between an output device and a system bus. It should be noted that other devices and / or systems of devices, such as a remote computer(s), may provide both input and output capabilities.

[0100] The computer system 20063 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computer(s), or local computers. The remote cloud computer(s) can be personal computers, servers, routers, network PCs, workstations, microprocessor-based appliances, peer devices, or other common network nodes, but typically include many or all of the elements described with respect to a computer system. For simplicity, only memory storage devices are shown along with the remote computer(s). The remote computer(s) may be logically connected to the computer system through a network interface and subsequently physically connected via a communications connection. The network interface may encompass communications networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, etc. WAN technologies may include, but are not limited to, point-to-point links, circuit-switched networks such as Integrated Services Digital Networks (ISDN) and its variations, packet-switched networks, and Digital Subscriber Lines (DSL).

[0101] In various examples, the computer system 20063 of Figures 4, 6A, and 6B, the imaging module 20088 and / or the human interface system 20080 of Figures 5 and 6A, and / or the processor module 20093 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used to process digital images. The image processor may employ parallel computing using single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0102] The communication connection(s) may refer to the hardware / software employed to connect the network interface to the bus. For illustrative clarity, the communication connections are shown internal to computer system 20063, however, the communication connections may be external to computer system 20063. By way of example only, the hardware / software required to connect to the network interface may include internal and external technologies such as modems, including regular telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0103] FIG. 6B illustrates an example of a wearable monitoring system, e.g., a control patient monitoring system. The control patient monitoring system may be a sensing system used to monitor a set of patient biomarkers while a patient is in a medical facility. The control patient monitoring system may be deployed for pre-operative patient monitoring when a patient is being prepared for a surgical procedure, intra-operative monitoring when a patient is undergoing surgery, or post-operative monitoring, for example, while a patient is recovering. As shown in FIG. 6B, the control patient monitoring system may include a surgical hub system 20076, which may include one or more routers 20066 and a computer system 20063 of a modular communications hub 20065. The router 20065 may include a wireless router, a wired switch, a wired router, a wired or wireless network hub, etc. In one example, the router 20065 may be part of an infrastructure. The computing system 20063 may provide local processing for monitoring various biomarkers associated with the patient or surgeon and a notification mechanism to indicate to the patient and / or healthcare provider (HCP) that milestones (e.g., recovery milestones) have been met or complications have been detected. The computing system 20063 of the surgical hub system 20076 may also be used to generate a severity level associated with a notification, for example, that a complication has been detected.

[0104] The computing system 20063 of Figures 4, 6B, the computing device 20200 of Figure 6C, or the hub / computing device 20243 of Figures 7B, 7C, or 7D may be a surgical computing system or hub device, a laptop, a tablet, a smartphone, or the like.

[0105] As shown in Figure 6B, a set of sensing systems 20069 and / or environmental sensing systems 20015 (as described in Figure 2A) may be connected to a surgical hub system 20076 via a router 20065. The router 20065 may also provide a direct communication connection between the sensing systems 20069 and a cloud computing system 20064, for example, without involving the local computer system 20063 of the surgical hub system 20076. Communication from the surgical hub system 20076 to the cloud 20064 may occur over either a wired or wireless communication channel.

[0106] As shown in FIG. 6B, computer system 20063 may include a processor 20102 and a network interface 20100. The processor 20102 may be coupled to a radio frequency (RF) interface or communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and input / output interface 20107 via a system bus, as described in FIG. 6A. Computer system 20063 may be connected to a local display unit 20108. In some examples, the display unit 20108 may be replaced by an HID. Details regarding the hardware and software components of the computer system are provided in FIG. 6A.

[0107] 6B, the sensing system 20069 may include a processor 20110. The processor 20110 may be coupled to a radio frequency (RF) interface 20114, storage 20113, memory (e.g., non-volatile memory) 20112, and an I / O interface 20111 via a system bus. The system bus may be any of several types of bus structure(s), including a memory bus or memory controller, a peripheral or external bus, and / or a local bus, as described herein. The processor 20110 may be any single-core or multi-core processor as described herein.

[0108] It should be understood that the sensing system 20069 may include software that acts as an intermediary between sensing system users and computer resources described in a suitable operating environment. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate computer system resources. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0109] The sensing system 20069 may be connected to a human interface system 20115. The human interface system 20115 may be a touchscreen display. The human interface system 20115 may include a human interface display to display information associated with surgeon biomarkers and / or patient biomarkers, display prompts for user actions by the patient or surgeon, or display notifications to the patient or surgeon indicating information regarding recovery milestones or complications. The human interface system 20115 may be used to receive input from the patient or surgeon. Other human interface systems may be connected to the sensing system 20069 via the I / O interface 20111. For example, the human interface device 20115 may include a device for providing tactile feedback as a mechanism to prompt the user to pay attention to notifications that may be displayed on the display unit.

[0110] The sensing system 20069 may operate in a network environment using logical connections to one or more remote computers, such as cloud computer(s) or a local computer. The remote cloud computer(s) may be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, or other common network node, but typically include many or all of the elements described with respect to a computer system. The remote computer(s) may be logically connected to the computer system through a network interface. The network interface may encompass communication networks such as a local area network (LAN), a wide area network (WAN), and / or a mobile network. LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, Wi-Fi / IEEE 802.11, etc. WAN technologies may include, but are not limited to, point-to-point links, circuit-switched networks such as Integrated Services Digital Networks (ISDN) and its variants, packet-switched networks, and Digital Subscriber Lines (DSL). A mobile network may include communication links based on one or more of the following mobile communication protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR) or 5G, etc.

[0111] 6C illustrates an exemplary non-control patient monitoring system, for example, when the patient is away from the medical facility. Non-control patient monitoring systems may be used for pre-operative patient monitoring, for example, when the patient is preparing for a surgical procedure but away from the medical facility, or for post-operative monitoring, for example, when the patient is recovering away from the medical facility.

[0112] 6C , one or more sensing systems 20069 communicate with a computing device 20200, such as a personal computer, laptop, tablet, or smartphone. The computing system 20200 may provide processing for monitoring various biomarkers associated with the patient, notifications indicating that milestones (e.g., recovery milestones) have been met, or complications have been detected. The computing system 20200 may also provide instructions for a user of the sensing system to follow. Communication between the sensing system 20069 and the computing device 20200 may be established directly using a wireless protocol as described herein or via a wireless router / hub 20211.

[0113] 6C , the sensing system 20069 may be connected to the computing device 20200 via a router 20211. The router 20211 may include a wireless router, a wired switch, a wired router, a wired or wireless network hub, etc. The router 20211 may, for example, provide a direct communication connection between the sensing system 20069 and the cloud server 20064 without involving the local computing device 20200. The computing device 20200 may communicate with the cloud server 20064. For example, the computing device 20200 may communicate with the cloud 20064 through a wired or wireless communication channel. In one example, the sensing system 20069 may communicate with the cloud directly through a cellular network, for example via a cellular base station 20210.

[0114] As shown in FIG. 6C , the computing device 20200 may include a processor 20203 and a network or RF interface 20201. The processor 20203 may be coupled to the storage 20202, memory 20212, non-volatile memory 20213, and input / output interface 20204 via a system bus, as described in FIGS. 6A and 6B . Details about the hardware and software components of the computer system are provided in FIG. 6A . The computing device 20200 may include a set of sensors, such as sensor #1 20205, sensor #2 20206, through sensor #n 20207. These sensors may be part of the computing device 20200 and may be used to measure one or more attributes associated with the patient. The attributes may provide context for biomarker measurements performed by one of the sensing systems 20069. For example, sensor #1 may be an accelerometer that may be used to measure acceleration forces to sense motion or vibration associated with the patient. In one example, the sensors 20205-20207 may include one or more of a pressure sensor, an altimeter, a thermometer, a lidar, and the like.

[0115] As shown in Figure 6B, the sensing system 20069 may include a processor, a radio frequency interface, storage, memory, or non-volatile memory, and an input / output interface via a system bus, as described in Figure 6A. The sensing system may include a sensor unit and a processing and communication unit, as described in Figures 7B through 7D. The system bus may be any of several types of bus structure(s), including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor may be any single-core or multi-core processor, as described herein.

[0116] The sensing system 20069 can communicate with a human interface system 20215. The human interface system 20215 may be a touchscreen display. The human interface system 20215 may be used to display information associated with patient biomarkers, display prompts for user actions by the patient, or display notifications to the patient indicating information regarding recovery milestones or complications. The human interface system 20215 may be used to receive input from the patient. Other human interface systems may be connected to the sensing system 20069 via an I / O interface. For example, the human interface system may include a device for providing tactile feedback as a mechanism to prompt the user to pay attention to notifications that may be displayed on the display unit. The sensing system 20069 may operate in a networked environment using logical connections to one or more remote computers, such as cloud computer(s) or a local computer, as described in FIG. 6B .

[0117] FIG. 7A shows a logic diagram of a surgical instrument or tool control system 20220 according to one or more embodiments of the present disclosure. The surgical instrument or tool may be configurable. The surgical instrument may include surgical fasteners specific to the procedure at hand, such as an imaging device, a surgical stapler, an energy device, an endocutter device, etc. For example, the surgical instrument may include any of a power stapler, a power stapler generator, an energy device, an advanced energy device, an advanced energy jaw device, an endocutter clamp, an energy device generator, an operating room imaging device, a smoke evacuation device, a suction irrigation device, an air delivery system, etc. The system 20220 may include control circuitry. The control circuitry may include a microcontroller 20221 with a processor 20222 and a memory 20223. For example, one or more of sensors 20225, 20226, 20227 provide real-time feedback to the processor 20222. A motor 20230, driven by a motor driver 20229, operatively couples the longitudinally movable displacement member to drive the I-beam knife element. A tracking system 20228 can be configured to determine the position of the longitudinally movable displacement member. The position information can be provided to a processor 20222, which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the firing member, firing bar, and I-beam knife element. Additional motors can be provided to the tool driver interface to control the firing of the I-beam, the movement of the obturator tube, the rotation of the shaft, and the articulation. A display 20224 can display various operating conditions of the instrument and can include touchscreen functionality for data entry. Information displayed on the display 20224 can be overlaid with images acquired via the endoscopic imaging module.

[0118] In one aspect, microcontroller 20221 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In one aspect, main microcontroller 20221 may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, including on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle SRAM, internal ROM loaded with StellarisWare® software, 2 KB of EEPROM, one or more PWM modules, one or more QEI analog, and / or one or more 12-bit ADCs with 12 analog input channels, details of which are available in the product datasheet.

[0119] In one aspect, the microcontroller 20221 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller may be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0120] The microcontroller 20221 may be programmed to perform various functions, such as precise control over the speed and position of the knife and articulation system. In one aspect, the microcontroller 20221 may include a processor 20222 and memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and mechanical link to the articulation or knife system. In one aspect, the motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be easily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of absolute positioning systems is provided in U.S. Patent Application Publication No. 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety.

[0121] The microcontroller 20221 may be programmed to provide precise control over the velocity and position of the displacement members and articulation system. The microcontroller 20221 may be configured to calculate a response within the microcontroller 20221 software. The calculated response may be compared to the measured response of the actual system to obtain an "observed" response, which is used to determine actual feedback. The observed response may be a suitably adjusted value that balances the smooth, continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0122] In some examples, the motor 20230 may be controlled by a motor driver 20229 and may be employed by the surgical instrument or tool firing system. In various forms, the motor 20230 may be a brushed DC drive motor having a maximum rotational speed of, for example, about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver including field effect transistors (FETs). The motor 20230 may be powered by a power supply assembly releasably attached to the handle assembly or tool housing to provide control power to the surgical instrument or tool. The power supply assembly may include a battery, which may include multiple battery cells connected in series, that may be used as a power source to power the surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be lithium ion batteries that may be connectable to and separable from the power supply assembly.

[0123] The motor driver 20229 may be the A3941, available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with external N-channel power metal-oxide semiconductor field-effect transistors (MOSFETs), specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include its own charge-pump regulator, capable of providing full (>10 V) gate drive for battery voltages up to 7 V, allowing the A3941 to operate with reduced gate drive down to 5.5 V. A bootstrap capacitor may be used to provide the required battery supply voltage above the N-channel MOSFET. An internal charge pump for the high-side drive allows DC (100% duty cycle) operation. The full-bridge may be driven in fast or slow decay mode using diode or synchronous rectification. In slow decay mode, current recirculation is possible through either the high-side or low-side FET. The power FETs may be protected from shoot-through by a resistor-adjustable dead time. Integrated diagnostics indicate undervoltage, overtemperature, and power bridge faults and can be configured to protect the power MOSFETs under most short circuit conditions. Other motor drivers may be easily substituted for use in tracking system 20228 with an absolute positioning system.

[0124] The tracking system 20228 may include a controlled motor drive circuit arrangement including a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member including a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a firing member that may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a firing bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may be used generally to refer to any movable member of a surgical instrument or tool, such as a drive member, firing member, firing bar, I-beam, or any element that can be displaced. In one aspect, a longitudinally movable drive member may be coupled to a firing member, firing bar, and I-beam. Thus, the absolute positioning system may, in effect, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member may be coupled to any position sensor 20225 suitable for measuring linear displacement. Thus, the longitudinally movable drive member, firing member, firing bar, or I-beam, or combinations thereof, may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor. The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system comprising a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0125] The electric motor 20230 may include a rotatable shaft operably interfaced with a gear assembly mounted in meshing engagement with a set of drive teeth or rack on the displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 20225 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor arrangement may be connected to a linear actuator by a rack and pinion arrangement or to a rotary actuator by a spur gear or other connection. A power source may provide power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable firing member, a firing bar, an I-beam, or a combination thereof.

[0126] One revolution of the sensor element associated with the position sensor 20225 may correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance traveled by the displacement member from point "a" to point "b" after one revolution of the sensor element coupled to the displacement member. The sensor arrangement may be coupled via a gear reduction that results in the position sensor 20225 completing one or more revolutions for a full stroke of the displacement member. The position sensor 20225 may complete multiple revolutions for a full stroke of the displacement member.

[0127] A series of switches may be used alone or in combination with gear reductions to provide unique position signals for two or more revolutions of the position sensor 20225, where n is an integer greater than 1. The states of the switches may be fed back to a microcontroller 20221 which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is provided to the microcontroller 20221. The position sensor 20225 in the sensor arrangement may comprise a magnetic sensor, an analog rotary sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0128] The position sensor 20225 may comprise any number of magnetic sensing elements, such as, for example, magnetic sensors classified by whether they measure the total magnetic field or vector components of the magnetic field. The technologies used to produce both types of magnetic sensors may involve many aspects of physics and electronics. Technologies used to sense magnetic fields may include, among others, search coils, fluxgates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetodiodes, magnetotransistors, fiber optics, magneto-optics, and microelectromechanical systems-based magnetic sensors.

[0129] In one aspect, the position sensor 20225 of the tracking system 20228 with an absolute positioning system may comprise a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 20225 is interfaced with the microcontroller 20221 to provide the absolute positioning system. The position sensor 20225 may be a low-voltage, low-power component and may include four Hall-effect elements in an area of ​​the position sensor 20225 that may be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on-chip. A Coordinate Rotation Digital Computer (CORDIC) processor, also known as the Digit-by-Digit Method and the Boulder algorithm, may be provided to implement simple and efficient algorithms for calculating hyperbolic and trigonometric functions, requiring only addition, subtraction, bit shifting, and table lookup operations. The angular position, alarm bits, and magnetic field information can be transmitted to the microcontroller 20221 via a standard serial communication interface, such as a serial peripheral interface (SPI) interface. The position sensor 20225 can provide 12-bit or 14-bit resolution. The position sensor 20225 can be an AS5055 chip, which comes in a small QFN 16-pin 4x4x0.85mm package.

[0130] A tracking system 20228 comprising an absolute positioning system may comprise and / or be programmed to implement a feedback controller, such as a PID, state feedback, and adaptive controller. A power supply converts a signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensor(s) may be provided to measure physical parameters of the physical system. In some embodiments, the other sensor(s) may include sensor arrangements such as those described in U.S. Pat. No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuitry to combine the calculated response with the measured response using algorithms such as weighted averages and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account properties such as mass, inertia, viscous friction, and induced drag in order to predict what the state and output of the physical system will be given knowledge of the input.

[0131] The absolute positioning system can provide the absolute position of the displacement member when the instrument is powered on without retracting or advancing the displacement member to a reset (zero or home) position, as may be required with conventional rotary encoders that simply count the number of forward or backward steps taken by the motor 20230 to estimate the position of the device actuator, drive bar, knife, etc.

[0132] The sensor 20226, such as a strain gauge or micro-strain gauge, can be configured to measure one or more parameters of the end effector, such as the amplitude of strain exerted on the anvil during clamping, which can represent the closure force applied to the anvil. The measured strain can be converted to a digital signal and provided to the processor 20222. Alternatively or in addition to the sensor 20226, a sensor 20227, such as a load sensor, can measure the closure force applied to the anvil by the closure drive system. The sensor 20227, such as a load sensor, can measure the firing force applied to the I-beam during the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sled, which is configured to cam the staple driver upward and drive the staples into deforming contact with the anvil. The I-beam can also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 20231 can be employed to measure the current drawn by the motor 20230. The force required to advance the firing member can correspond, for example, to the current drawn by the motor 20230. The measured force can be converted to a digital signal and provided to the processor 20222.

[0133] In one form, a strain gauge sensor 20226 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue grasped by the end effector can include a strain gauge sensor 20226, such as a micro strain gauge, configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 20226 can measure the amplitude or magnitude of strain exerted on the jaw members of the end effector during clamping, which can indicate tissue compression. The measured strain can be converted to a digital signal and provided to the processor 20222 of the microcontroller 20221. A load sensor 20227 can measure the force used to operate the knife element, for example, to cut tissue captured between the anvil and the staple cartridge. A magnetic field sensor can be used to measure the thickness of the captured tissue. The magnetic field sensor's measurements can also be converted to a digital signal and provided to the processor 20222.

[0134] Measurements of tissue compression, tissue thickness, and / or force required to close the end effector on the tissue, measured by sensors 20226, 20227, respectively, can be used by microcontroller 20221 to characterize a selected position of the firing member and / or a corresponding value of firing member velocity. In one example, memory 20223 can store techniques, equations, and / or look-up tables that can be employed by microcontroller 20221 in the evaluation.

[0135] The surgical instrument or tool control system 20220 may also include wired or wireless communication circuitry for communicating with the modular communication hub 20065, as shown in Figures 5 and 6A.

[0136] 7B shows an exemplary sensing system 20069. The sensing system may be a surgeon sensing system or a patient sensing system. The sensing system 20069 may include a sensor unit 20235 and a human interface system 20242 in communication with a data processing and communication unit 20236. The data processing and communication unit 20236 may include an analog-to-digital conversion 20237, a data processing unit 20238, a storage unit 20239, an input / output interface 20241, and a transceiver 20240. The sensing system 20069 may communicate with a surgical hub or computing device 20243, which communicates with a cloud computing system 20244. The cloud computing system 20244 may include a cloud storage system 20078 and one or more cloud servers 20077.

[0137] The sensor unit 20235 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers, such as blood pH, hydration status, oxygen saturation, core body temperature, heart rate, heart rate variability, sweat rate, skin conductance, blood pressure, light exposure, ambient temperature, respiratory rate, coughing and sneezing, gastrointestinal motility, gastrointestinal imaging, tissue perfusion pressure, airway bacteria, alcohol consumption, lactic acid (sweat), peripheral temperature, positivity and optimism, adrenaline (sweat), cortisol (sweat), edema, mycotoxins, VO2 max, pre-operative pain, airborne chemicals, circulating tumor cells, stress and anxiety, confusion and delirium, physical activity, autonomic tone, circadian rhythm, menstrual cycle, sleep, etc. These biomarkers may be measured using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The sensors may measure the biomarkers described herein using one or more of the following sensing technologies: photoplethysmography, electrocardiography, electroencephalography, colorimetry, obstruction testing, potentiometry, amperometry, etc.

[0138] 7B , the sensors in the sensor unit 20235 may measure physiological signals (e.g., voltage, current, PPG signal, etc.) associated with the biomarker being measured. The physiological signals measured may depend on the sensing technology used, as described herein. The sensor unit 20235 of the sensing system 20069 may communicate with the data processing and communication unit 20236. In one example, the sensor unit 20235 may communicate with the data processing and communication unit 20236 using a wireless interface. The data processing and communication unit 20236 may include an analog-to-digital converter (ADC) 20237, a data processing unit 20238, storage 20239, an I / O interface 20241, and an RF transceiver 20240. The data processing unit 20238 may include a processor and a memory unit.

[0139] The sensor unit 20235 may transmit the measured physiological signals to the ADC 20237 of the data processing and communication unit 20236. In one example, the measured physiological signals may be passed through one or more filters (e.g., an RC low-pass filter) before being transmitted to the ADC. The ADC may convert the measured physiological signals into measurement data associated with the biomarkers. The ADC may pass the measurement data to the data processing unit 20238 for processing. In one example, the data processing unit 20238 may transmit the measurement data associated with the biomarkers to a surgical hub or computing device 20243, which may then transmit the measurement data to a cloud computing system 20244 for further processing. The data processing unit may transmit the measurement data to the surgical hub or computing device 20243 using one of the wireless protocols as described herein. In one example, the data processing unit 20238 may first process the raw measurement data received from the sensor unit and transmit the processed measurement data to the surgical hub or computing device 20243.

[0140] In one example, the data processing and communication unit 20236 of the sensing system 20069 may receive threshold values ​​associated with biomarkers from the surgical hub, the computing device 20243, or directly from the cloud server 20077 of the cloud computing system 20244. The data processing unit 20236 may compare measurement data associated with the monitored biomarkers with the corresponding threshold values ​​received from the surgical hub, the computing device 20243, or the cloud server 20077. The data processing and communication unit 20236 may send a notification message to the HID 20242 indicating that the measurement data value has exceeded the threshold value. The notification message may include the measurement data associated with the monitored biomarker. The data processing and computing unit 20236 may send the notification via transmission to the surgical hub or the computing device 20243 using one of the following RF protocols: Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Wi-Fi. The data processing unit 20238 may send notifications (e.g., notifications from the HCP) directly to a cloud server via transmission to a cellular transmission / reception point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR) or 5G. In one example, the sensing unit may communicate with a hub / computing device through a router, as described in Figures 6A-6C.

[0141] 7C shows an exemplary sensing system 20069 (e.g., a surgeon sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20245, a data processing and communication unit 20246, and a human interface device 20242. The sensor unit 20245 may include a sensor 20247 and an analog-to-digital converter (ADC) 20248. The ADC 20248 in the sensor unit 20245 may convert physiological signals measured by the sensor 20247 into measurement data associated with biomarkers. The sensor unit 20245 may transmit the measurement data to the data processing and communication unit 20246 for further processing. In one example, the sensor unit 20245 may transmit the measurement data to the data processing and communication unit 20246 using an inter-integrated circuit (I2C) interface.

[0142] The data processing and communication unit 20246 includes a data processing unit 20249, a storage unit 20250, and an RF transceiver 20251. The sensing system may be in communication with a surgical hub or computing device 20243, which may be in communication with a cloud computing system 20244. The cloud computing system 20244 may include a remote server 20077 and associated remote storage 20078. The sensor unit 20245 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers as described herein.

[0143] The data processing and communication unit 20246, after processing the measurement data received from the sensor unit 20245, may further process the measurement data and / or send the measurement data to a smart device or computing device 20243, as described in Figure 7B. In one example, the data processing and communication unit 20246 may send the measurement data received from the sensor unit 20245 to a remote server 20077 in a cloud computing system 20244 for further processing and / or monitoring.

[0144] 7D shows an exemplary sensing system 20069 (e.g., a surgeon sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20252, a data processing and communication unit 20253, and a human interface system 20261. The sensor unit 20252 may include multiple sensors 20254, 20255, through 20256 to measure one or more physiological signals associated with biomarkers of the patient or surgeon and / or one or more physical condition signals associated with the physical condition of the patient or surgeon. The sensor unit 20252 may also include one or more analog-to-digital converters (ADCs) 20257. The list of biomarkers may include biomarkers such as those disclosed herein. The ADC(s) 20257 in the sensor unit 20252 may convert each of the physiological signals and / or physical condition signals measured by the sensors 20254-20256 into respective measurement data. The sensor unit 20252 may transmit measurement data associated with one or more biomarkers and the physical condition of the patient or surgeon to the data processing and communication unit 20253 for further processing. The sensor unit 20252 may transmit the measurement data for each of the sensors Sensor 1 20254 to Sensor N 20256 individually or combined for all sensors to the data processing and communication unit 20253. In one example, the sensor unit 20252 may transmit the measurement data to the data processing and communication unit 20253 via an I2C interface.

[0145] The data processing and communication unit 20253 may include a data processing unit 20258, a storage unit 20259, and an RF transceiver 20260. The sensing system 20069 may be in communication with a surgical hub or computing device 20243, which is in communication with a cloud computing system 20244 comprising at least one remote server 20077 and at least one storage unit 20078. The sensor unit 20252 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers as described herein.

[0146] FIG. 8 illustrates an example of using surgical task context awareness and measurement data from one or more surgeon sensing systems to adjust surgical instrument control. FIG. 8 illustrates a timeline 20265 of an exemplary surgical procedure and context information that the surgical hub can derive from data received from one or more surgical devices, one or more surgeon sensing systems, and / or one or more environmental sensing systems at each step of the surgical procedure. Devices that can be controlled by the surgical hub may include advanced energy devices, endocutter clamps, etc. The surgeon sensing system may include a sensing system for measuring one or more biomarkers associated with the surgeon, such as heart rate, sweat composition, respiratory rate, etc. The environmental sensing system may include a system for measuring one or more environmental attributes, such as a camera for detecting the surgeon's position / movement / breathing patterns, a spatial microphone for measuring ambient noise within the surgical field and / or the tone of a clothing provider's voice, ambient temperature / humidity, etc.

[0147] In the following description of the timeline 20265 shown in FIG. 8, please also refer to FIG. 5. FIG. 5 provides various components used in a surgical procedure. The timeline 20265 illustrates steps that may be performed individually or collectively by nurses, surgeons, and other medical personnel during an exemplary colorectal surgical procedure. In a colorectal surgical procedure, the situational awareness surgical hub 20076 may receive data from various data sources throughout the course of the surgical procedure, including data generated each time a healthcare provider (HCP) utilizes a modular device / instrument 20095 paired with the surgical hub 20076. The surgical hub 20076 may receive this data from the paired modular device 20095. The surgical hub may receive measurement data from a sensing system 20069. The surgical hub may use data from the modular devices / instruments 20095 and / or measurement data from the sensing system 20069 to continuously derive inferences (i.e., contextual information) regarding the HCP's stress level and the ongoing procedure as new data is received, such that the surgeon's stress level relative to the procedure step being performed is derived. The situational awareness system of the surgical hub 20076 may perform one or more of the following: record data regarding the procedure to generate a report, verify steps being taken by the medical personnel, provide data or prompts (e.g., via a display screen) that may be relevant to a particular procedure step, adjust the modular device based on the context (e.g., activate a monitor, adjust the FOV of a medical imaging device, change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), or perform other such actions described herein. In one example, these steps may be performed by a remote server 20077 of the cloud system 20064, which may communicate with the surgical hub 20076.

[0148] As a first step (not shown in FIG. 8 for simplicity), hospital personnel may retrieve the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hub 20076 may determine that the procedure to be performed is a colorectal procedure. The personnel may scan the incoming medical supplies for the procedure. The surgical hub 20076 may cross-reference the scanned supplies with a list of supplies available for various types of procedures and confirm that the combination of supplies may correspond to a colorectal procedure. The surgical hub 20076 may detect each of the sensing systems 20069 worn by different HCPs.

[0149] Once each of the devices is prepared and pre-operative preparation is complete, the surgical team can begin by making an incision and placing a trocar. The surgical team can access and prepare by dissecting adhesions, if any, and identifying the inferior mesenteric artery (IMA) branch. The surgical hub 20076 can infer that the surgeon is in the process of dissecting adhesions based on data that may be received from at least the RF or ultrasonic generator indicating that an energy instrument is being fired. The surgical hub 20076 can cross-reference the received data with the acquired steps of the surgical procedure to determine that the energy instrument being fired at this point in the process (e.g., after completion of the procedure steps discussed above) corresponds to the incision step.

[0150] After incision, the HCP can proceed to the ligation step of the procedure (e.g., indicated by A1). As shown in FIG. 8 , the HCP can begin by ligating the IMA. The surgical hub 20076 can infer that the surgeon is ligating the artery and vein because it can receive data from the advanced energy jaw device and / or endocutter indicating that the instrument is being fired. The surgical hub can also receive measurement data from one of the HCP's sensing systems indicating a higher stress level for the HCP (e.g., indicated by the B1 mark on the time axis). For example, a higher stress level may be indicated by a change in the HCP's heart rate from a baseline. As with the previous step, the surgical hub 20076 can derive this inference by cross-referencing the receipt of data from the surgical stapling and severing instrument with the acquired steps in the process (e.g., as indicated by A2 and A3). The surgical hub 20076 can monitor the advanced energy jaw trigger ratio and / or endocutter clamp and firing rate during periods of high stress. In one example, the surgical hub 20076 may send assistance control signals to the advanced energy jaw device and / or the endocutter clamp to control the device during operation. The surgical hub may send assistance signals based on the stress level of the HCP operating the surgical device and / or situational awareness known to the surgical hub. For example, the surgical hub 20076 may send control assistance signals to the advanced energy device or the endocutter clamp, as shown in FIG. 8 by A2 and A3.

[0151] The HCP can proceed to the next step of releasing the upper sigmoid colon, followed by releasing the descending colon, rectum, and sigmoid colon. The surgical hub 20076 can continue to monitor the HCP's high stress markers (e.g., as indicated by D1, E1a, E1b, F1). The surgical hub 20076 can send an assist signal to the advanced energy jaw device and / or endocutter device during periods of high stress, as shown in FIG. 8.

[0152] After mobilizing the colon, the HCP can proceed with the segmental resection portion of the procedure. For example, the surgical hub 20076 can estimate that the HCP is resecting the intestine and removing the sigmoid colon based on data from the surgical stapling and severing instrument, including data from its cartridge. The cartridge data can correspond, for example, to the size or type of staples being fired by the instrument. Because different types of staples are utilized for different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transected. Note that surgeons regularly switch between surgical stapling / severing instruments and surgical energy (e.g., RF or ultrasonic) instruments depending on the step in the procedure, as different instruments are better suited for specific tasks. Thus, the sequence in which the stapling / severing instrument and surgical energy instrument are used can indicate which step of the procedure the surgeon is performing.

[0153] The surgical hub may determine and send a control signal to the surgical device based on the HCP's stress level. For example, control signal G2b may be sent to the endocutter clamp during period G1b. Once the sigmoid colon is removed, the incision may be closed and the post-operative portion of the procedure may begin. The patient may emerge from anesthesia. The surgical hub 20076 may estimate that the patient is cooling down from anesthesia based on one or more sensing systems attached to the patient.

[0154] 9 is a block diagram of a computer-implemented interactive surgical system with surgeon / patient monitoring according to at least one embodiment of the present disclosure. In one embodiment, the computer-implemented interactive surgical system can be configured to monitor surgeon biomarkers and / or patient biomarkers using one or more sensing systems 20069. The surgeon biomarkers and / or patient biomarkers can be measured before, after, and / or during a surgical procedure. In one embodiment, the computer-implemented interactive surgical system can be configured to monitor and analyze data related to the operation of various surgical systems 20069, including surgical hubs, surgical instruments, robotic devices, and surgical sites or medical facilities. The computer-implemented interactive surgical system can include a cloud-based analysis system. The cloud-based analysis system can include one or more analysis servers.

[0155] 9, the cloud-based monitoring and analysis system may include a plurality of sensing systems 20268 (which may be the same as or similar to sensing system 20069), a surgical instrument 20266 (which may be the same as or similar to instrument 20031), a plurality of surgical hubs 20270 (which may be the same as or similar to hub 20006), and a surgical data network 20269 (which may be the same as or similar to the surgical data network shown in FIG. 4) for connecting the surgical hubs 20270 to a cloud 20271 (which may be the same as or similar to cloud computing system 20064). Each of the plurality of surgical hubs 20270 may be communicatively coupled to one or more surgical instruments 20266. Each of the plurality of surgical hubs 20270 may also be communicatively coupled to one or more sensing systems 20268 and to the cloud of computer-implemented interactive surgical systems 20271 via network 20269. The surgical hub 20270 and the sensing system 20268 may be communicatively coupled using a wireless protocol as described herein. The cloud system 20271 may be a remote, centralized source of hardware and software for storing, processing, manipulating, and communicating measurement data from the sensing system 20268 and data generated based on the various surgical systems 20268.

[0156] As shown in FIG. 9 , access to the cloud system 20271 can be achieved via a network 20269, which may be the Internet or some other suitable computer network. A surgical hub 20270, which may be coupled to the cloud system 20271, can be considered the client side of a cloud computing system (e.g., a cloud-based analysis system). Surgical instruments 20266 may be paired with the surgical hub 20270 for control and implementation of various surgical procedures and / or operations, as described herein. A sensing system 20268 may be paired with the surgical hub 20270 for intraoperative surgeon monitoring of surgeon-related biomarkers, preoperative patient monitoring, intraoperative patient monitoring, or postoperative monitoring of patient biomarkers to track and / or measure various milestones and / or detect various complications. An environmental sensing system 20267 may be paired with the surgical hub 20270 to measure environmental attributes associated with the surgeon or patient for surgeon monitoring, preoperative patient monitoring, intraoperative patient monitoring, or postoperative patient monitoring.

[0157] The surgical instruments 20266, environmental sensing system 20267, and sensing system 20268 may include wired or wireless transceivers for data transmission to and from their corresponding surgical hub 20270 (which may also include a transceiver). The combination of one or more of the surgical instruments 20266, sensing system 20268, or surgical hub 20270 may indicate a specific location, such as a surgical site, an intensive care unit (ICU) room, or a recovery room, within a medical facility (e.g., a hospital) to provide for medical surgery, pre-operative preparation, and / or post-operative recovery. For example, a memory of the surgical hub 20270 may store location data.

[0158] 9, cloud system 20271 may include one or more central servers 20272 (which may be the same as or similar to remote server(s) 20067), a surgical hub application server 20276, a data analysis module 20277, and an input / output ("I / O") interface 20278. The central servers 20272 of cloud system 20271 may collectively manage the cloud computing system, including monitoring requests by client surgical hubs 20270 and managing the processing power of the cloud system 20271 to carry out the requests. Each of the central servers 20272 may include one or more processors 20273 coupled to suitable memory devices 20274, which may include volatile memory, such as random access memory (RAM), and non-volatile memory, such as magnetic storage devices. The memory device 20274 may include machine-executable instructions that, when executed, cause the processor 20273 to execute a data analysis module 20277 for cloud-based data analysis, real-time monitoring of measurement data received from the sensing system 20268, action, recommendations, and other operations described herein. The processor 20273 may execute the data analysis module 20277 independently or in conjunction with a hub application executed independently by the hub 20270. The central server 20272 may also include a database 20275 of aggregated medical data, which may reside in the memory 20274.

[0159] Based on its connection to various surgical hubs 20270 via network 20269, cloud 20271 can aggregate data from particular data generated by various surgical instruments 20266 and / or monitor real-time data from sensing systems 20268 and surgical instruments 20266 and / or surgical hubs 20270 associated with sensing systems 20268. Such aggregated data from surgical instruments 20266 and / or measurement data from sensing systems 20268 may be stored in an aggregated medical database 20275 of cloud 20271. Specifically, cloud 20271 can advantageously track real-time measurement data from sensing systems 20268 and / or perform data analysis and actions on the measurement data and / or aggregated data to provide insights and / or perform functions that individual hubs 20270 cannot accomplish by themselves. To this end, as shown in FIG. 9 , cloud 20271 and surgical hubs 20270 are communicatively coupled to send and receive information. The I / O interface 20278 is connected to multiple surgical hubs 20270 via the network 20269. In this manner, the I / O interface 20278 can be configured to transfer information between the surgical hubs 20270 and the database of aggregated medical data 20275. Accordingly, the I / O interface 20278 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hubs 20270. These requests may be sent to the surgical hubs 20270 through a hub application. The I / O interface 20278 may include one or more high-speed data ports, which may include a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 20271 to the surgical hubs 20270. A hub application server 20276 of the cloud 20271 may be configured to host and serve sharing functionality to software applications (e.g., hub applications) executed by the surgical hubs 20270.For example, the hub application server 20276 may manage requests made by hub applications through the hub 20270, control access to the database of aggregated medical data 20275, and perform load balancing.

[0160] The cloud computing system configurations described in this disclosure may be designed to address a variety of problems that arise in the context of medical procedures (e.g., pre-operative monitoring, intra-operative monitoring, and post-operative monitoring) and procedures performed using medical devices such as surgical instruments 20266, 20031. Specifically, the surgical instruments 20266 may be digital surgical devices configured to interact with the cloud 20271 to implement techniques for improving surgical outcomes. The sensing system 20268 may be a system having one or more sensors configured to measure one or more biomarkers associated with the surgeon performing the medical procedure and / or the patient on whom the medical procedure will be, is being, or has been performed. The various surgical instruments 20266, sensing systems 20268, and / or surgical hub 20270 may include human interface systems (e.g., having touch-controlled user interfaces) so that a clinician and / or patient may control aspects of the interaction between the surgical instruments 20266 or sensing systems 20268 and the cloud 20271. Other suitable user interfaces for control may also be used, such as an audible controlled user interface.

[0161] The cloud computing system configurations described in this disclosure may be designed to address a variety of issues that arise in the context of using a sensing system 20268 to monitor one or more biomarkers associated with a healthcare professional (HCP) or a patient pre-operatively, intra-operatively, and post-surgically. The sensing system 20268 may be a surgeon sensing system or a patient sensing system configured to interact with a surgical hub 20270 and / or a cloud system 20271 to implement techniques for monitoring surgeon biomarkers and / or patient biomarkers. The various sensing systems 20268 and / or surgical hub 20270 may include touch-controlled human interface systems so that the HCP or patient may control aspects of the interaction between the sensing system 20268 and the surgical hub 20270 and / or cloud system 20271. Other suitable user interfaces for control, such as an audio-controlled user interface, may also be used.

[0162] 10 illustrates an exemplary surgical system 20280 according to the present disclosure and may include a surgical instrument 20282 that can communicate with a console 20294 or a portable device 20296 through a local area network 20292 or a cloud network 20293 via a wired or wireless connection. In various aspects, the console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 releasably couples to the handle 20297, and the loading unit 20287 releasably couples to the adapter 20285 such that the adapter 20285 transfers force from the drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein to measure force applied to the loading unit 20287. The loading unit 20287 can include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 can be an in-situ loading or multi-fire loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 20287 from the surgical site to reload the loading unit 20287.

[0163] The first and second jaws 20291, 20290 can be configured to clamp tissue therebetween, fire fasteners through the clamped tissue, and sever the clamped tissue. The first jaw 20291 can be configured to fire at least one fastener multiple times or can be configured to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that can be fired two or more times before being replaced. The second jaw 20290 can include an anvil that deforms or otherwise secures fasteners as they are ejected from the multi-fire fastener cartridge.

[0164] The handle 20297 can include a motor coupled to the drive shaft to affect rotation of the drive shaft. The handle 20297 can include a control interface for selectively activating the motor. The control interface can include buttons, switches, levers, sliders, a touch screen, and any other suitable input mechanism or user interface that can be engaged by a clinician to activate the motor.

[0165] The control interface of the handle 20297 may communicate with a controller 20298 of the handle 20297 to selectively activate the motor to affect rotation of the drive shaft. The controller 20298 may be provided within the handle 20297 and may be configured to receive input from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 may analyze the input from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display viewable by a clinician while using the handle 20297. The display may be configured to display a portion of the adapter or loading unit data before, during, or after firing of the instrument 20282.

[0166] The adapter 20285 may include an adapter identification device 20284 disposed therein, and the loading unit 20287 may include a loading unit identification device 20288 disposed therein. The adapter identification device 20284 may be in communication with a controller 20298, and the loading unit identification device 20288 may be in communication with the controller 20298. It will be appreciated that the loading unit identification device 20288 may be in communication with the adapter identification device 20284, which relays or passes through communications from the loading unit identification device 20288 to the controller 20298.

[0167] The adapter 20285 may also include a number of sensors 20286 (one shown) disposed about its periphery to detect various conditions of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to a loading unit, whether the adapter 20285 is connected to a handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain on the drive shaft, the temperature within the adapter 20285, the number of times the adapter 20285 has been fired, the peak force of the adapter 20285 during firing, the total amount of force applied to the adapter 20285, the peak retract force of the adapter 20285, the number of times the adapter 20285 has been paused during firing, etc.). The number of sensors 20286 can provide input to the adapter identification device 20284 in the form of data signals. The data signals of the number of sensors 20286 may be stored in the adapter identification device 20284 or may be used to update the adapter data stored in the adapter identification device 20284. The data signals of the plurality of sensors 20286 may be analog or digital. The plurality of sensors 20286 may include a force gauge for measuring the force applied to the loading unit 20287 during firing.

[0168] The handle 20297 and adapter 20285 can be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally, or alternatively, the electrical interface may be a contactless electrical interface for wirelessly transmitting energy and signals therebetween (e.g., inductively). It is also contemplated that the adapter identification device 20284 and the controller 20298 may wirelessly communicate with each other via a wireless connection that is separate from the electrical interface.

[0169] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., the LAN 20292, the cloud 20293, the console 20294, or the portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub 20270, as shown in FIG. 9 . The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 20280. For example, the controller 20298 may transmit instrument data to the console 20294 including the serial number of the attached adapter (e.g., adapter 20285) attached to the handle 20297, the serial number of the loading unit (e.g., loading unit 20287) attached to the adapter 20285, and the serial number of the multi-fire fastener cartridge loaded in the loading unit. The console 20294 may then transmit data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, back to the controller 20298. The controller 20298 can transmit a message to the console 20294 or portable device 20296 via the transceiver 20283 for displaying the message on a local instrument display or for displaying the message on the display 20295 or portable device screen, respectively.

[0170] 11A-11D show examples of wearable sensing systems, such as a surgeon sensing system or a patient sensing system. FIG. 11A is an example of an eyeglass-based sensing system 20300 that may be based on an electrochemical sensing platform. The sensing system 20300 may be capable of monitoring (e.g., real-time monitoring) sweat electrolytes and / or metabolites using multiple sensors 20304 and 20305 in contact with the surgeon's or patient's skin. For example, the sensing system 20300 may use an amperometric-based biosensor 20304 and / or a potentiometric-based biosensor 20305 integrated with the nose bridge pad of the eyeglasses 20302 to measure current and / or voltage.

[0171] The amperometric biosensor 20304 may be used to measure sweat lactate levels (e.g., in mmol / L). Lactate is a product of lactic acidosis, which can occur due to decreased tissue oxygenation, which can be caused by sepsis or hemorrhage. A patient's lactate level (e.g., >2 mmol / L) may be used to monitor the onset of sepsis, for example, during post-operative monitoring. The potentiometric biosensor 20305 may be used to measure a patient's sweat potassium level. A voltage follower circuit with an operational amplifier may be used to measure the potential signal between the reference electrode and the working electrode. The output of the voltage follower circuit may be filtered and converted to a digital value using an ADC.

[0172] The amperometric sensor 20304 and potentiometric sensor 20305 may be connected to a circuit 20303 located on each of the eyeglass arms. Electrochemical sensors may be used for simultaneous real-time monitoring of sweat lactate and potassium levels. Electrochemical sensors may be screen-printed on stickers and placed on both sides of the eyeglass nose pads to monitor sweat metabolites and electrolytes. The electronic circuit 20303 located on the eyeglass frame arms may include a wireless data transceiver (e.g., a low-energy Bluetooth transceiver) that can be used to transmit lactate and / or potassium measurement data to a surgical hub or intermediate device, which can then forward the measurement data to the surgical hub. The eyeglass-based sensing system 20300 may use a signal conditioning unit to filter and amplify the electrical signal generated from the electrochemical sensor 20305 or 20304, a microcontroller to digitize the analog signal, and a wireless (e.g., low-energy Bluetooth) module to forward the data to a surgical hub or computing device, as illustrated, for example, in FIGS. 7B-7D .

[0173] FIG. 11B illustrates an example of a wristband-type sensing system 20310 including a sensor assembly 20312 (e.g., a photoplethysmography (PPG)-based sensor assembly or an electrocardiogram (ECG)-based sensor assembly). For example, in the sensing system 20310, the sensor assembly 20312 may collect and analyze arterial pulses at the wrist. The sensor assembly 20312 may be used to measure one or more biomarkers (e.g., heart rate, heart rate variability (HRV), etc.). In a sensing system including a PPG-based sensor assembly 20312, light (e.g., green light) may be transmitted through the skin. A percentage of the green light may be absorbed by blood vessels, and some of the green light may be reflected and detected by a photodetector. These differences or reflections are associated with variations in blood perfusion in the tissue, which may be used to detect heart-related information (e.g., heart rate) of the cardiovascular system. For example, the amount of absorption may vary depending on the blood volume. The sensing system 20310 may determine heart rate by measuring light reflectance as a function of time. HRV may be determined as the variation (e.g., standard deviation) between the steepest signal peaks before the peak, known as the interbeat interval (IBI).

[0174] In the case of a sensing system having an ECG-based sensor assembly 20312, a set of electrodes may be placed in contact with the skin. The sensing system 20310 may measure the voltage across the set of electrodes placed on the skin to determine heart rate. HRV in this case may be measured as the period variation (e.g., standard deviation) between the R peaks of the QRS complex, known as the RR interval.

[0175] The sensing system 20310 can use a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., Bluetooth) module for transferring data to a surgical hub or computing device, for example, as described in Figures 7B-7D.

[0176] FIG. 11C illustrates an exemplary ring sensing system 20320. The ring sensing system 20320 may include a sensor assembly (e.g., a heart rate sensor assembly) 20322. The sensor assembly 20322 may include a light source (e.g., a red or green light emitting diode (LED)) and a photodiode for detecting reflected and / or absorbed light. The LED in the sensor assembly 20322 can shine light through the finger, and the photodiode in the sensor assembly 20322 can measure heart rate and / or blood oxygen levels by detecting blood volume changes. The ring sensing system 20320 may also include other sensor assemblies for measuring other biomarkers, such as a thermistor or infrared thermometer for measuring body surface temperature. The ring sensing system 20320 may use a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., low-energy Bluetooth) module for transferring data to a surgical hub or computing device, as illustrated in FIGS. 7B-7D .

[0177] FIG. 11D illustrates an example of an electroencephalogram (EEG) sensing system 20315. As shown in FIG. 11D, the sensing system 20315 may include one or more EEG sensor units 20317. The EEG sensor unit 20317 may include multiple conductive electrodes placed in contact with the scalp. The conductive electrodes may be used to measure small electrical potentials that may arise outside the head due to neuronal activity in the brain. The EEG sensing system 20315 can measure biomarkers, such as delirium, by identifying specific brain patterns, such as slowing or loss of occipital-dominant rhythms and loss of responsiveness to eye opening and closing. The ring sensing system 20315 may include a signal conditioning unit for filtering and amplifying the electrical potentials, a microcontroller for digitizing the electrical signal, and a wireless (e.g., low-energy Bluetooth) module for transferring data to a smart device, as illustrated in FIGS. 7B-7D.

[0178] FIG. 12 shows a block diagram of a computer-implemented patient / surgeon monitoring system 20325 for monitoring one or more patient or surgeon biomarkers before, during, and / or after a surgical procedure. As shown in FIG. 12, one or more sensing systems 20336 can be used to measure and monitor patient biomarkers, for example, to facilitate patient preparation before a surgical procedure and recovery after a surgical procedure. The sensing system 20336 can be used to measure and monitor surgeon biomarkers in real time to assist in surgical tasks, for example, by communicating associated biomarkers (e.g., surgeon biomarkers) to the surgical hub 20326 and / or surgical devices 20337 to adjust the functionality of the surgical hub 20326 and / or surgical devices. Surgical device functions that can be adjusted can include power level, advancement speed, closure speed, load, wait time, or other tissue-dependent operating parameters. The sensing system 20336 can also measure one or more physical attributes associated with the surgeon or patient. Patient biomarkers and / or physical attributes can be measured in real time.

[0179] The computer-implemented wearable patient / surgeon-wearable sensing system 20325 may include a surgical hub 20326, one or more sensing systems 20336, and one or more surgical devices 20337. The sensing systems and surgical devices may be communicatively coupled to the surgical hub 20326. One or more analysis servers 20338, for example, part of an analysis system, may also be communicatively coupled to the surgical hub 20326. While a single surgical hub 20326 is shown, it should be noted that the wearable patient / surgeon-wearable sensing system 20325 may include any number of surgical hubs 20326, which may be connected to form a network of surgical hubs 20326 that are communicatively coupled to one or more analysis servers 20338, as described herein.

[0180] In one example, the surgical hub 20326 may be a computing device. The computing device may be a personal computer, a laptop, a tablet, a smart mobile device, etc. In one example, the computing device may be a client computing device of a cloud-based computing system. The client computing device may be a thin client.

[0181] In one example, the surgical hub 20326 may include a processor 20327 coupled to a memory 20330 for executing stored instructions, storage 20331 for storing one or more databases, such as an EMR database, and a data relay interface 20329 through which data is transmitted to an analytics server 20338. In one example, the surgical hub 20326 may further include an I / O interface 20333 having an input device 20341 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 20335 (e.g., a display screen) for providing output to the user. In one example, the input device and output device may be a single device. The output may include data from a query entered by a user, products or product combinations for use in a given procedure, and / or instructions for actions to be performed before, during, and / or after the surgical procedure. The surgical hub 20326 may include a device interface 20332 for communicatively coupling a surgical device 20337 to the surgical hub 20326. In one aspect, the device interface 20332 may include a transceiver that may allow one or more surgical devices 20337 to connect with the surgical hub 20326 via a wired or wireless interface using one of the wired or wireless communication protocols described herein. The surgical devices 20337 may include, for example, a power stapler, an energy device or generator thereof, an imaging device, or other coupled systems such as a smoke evacuator, a suction irrigation device, an air insufflation system, etc.

[0182] In one example, the surgical hub 20326 may be communicatively coupled to one or more surgeon and / or patient sensing systems 20336. The sensing systems 20336 may be used to measure and / or monitor, in real time, various biomarkers associated with the surgeon performing the surgical procedure or the patient on whom the surgical procedure is being performed. A list of patient / surgeon biomarkers measured by the sensing systems 20336 is provided herein. In one example, the surgical hub 20326 may be communicatively coupled to an environmental sensing system 20334. The environmental sensing system 20334 may be used to measure and / or monitor, in real time, environmental attributes, such as temperature / humidity within the surgical field, surgeon movement, ambient noise within the surgical field caused by the surgeon's and / or patient's breathing patterns, etc.

[0183] When the sensing system 20336 and / or surgical device 20337 are connected to the surgical hub 20326, the surgical hub 20326 can receive measurement data associated with one or more patient biomarkers, a physical condition associated with the patient, measurement data associated with surgeon biomarkers, and / or a physical condition associated with the surgeon from the sensing system 20336, for example, as shown in Figures 7B-7D. The surgical hub 20326 can associate the measurement data associated with the surgeon, for example, with other relevant pre-operative data and / or data from the situational awareness system to generate control signals for controlling the surgical device 20337, for example, as shown in Figure 8.

[0184] In one example, the surgical hub 20326 may compare measurement data from the sensing system 20336 to one or more thresholds defined based on baseline values, pre-operative measurement data, and / or intra-operative measurement data. The surgical hub 20326 may compare measurement data from the sensing system 20336 to one or more thresholds in real time. The surgical hub 20326 may generate a notification for display. The surgical hub 20326 may send a notification for delivery to a human interface system for the patient 20339 and / or a human interface system for the surgeon or HCP 20340, for example, if the measurement data exceeds (e.g., is greater than or less than) a defined threshold. The determination of whether a notification is sent to one or more of the human interface systems for the patient 20339 and / or the HCP 2340 may be based on a severity level associated with the notification. The surgical hub 20326 may also generate a severity level associated with the notification for display. The generated severity level may be displayed to the patient and / or the surgeon or HCP. In one example, patient biomarkers measured and / or monitored (e.g., measured and / or monitored in real time) may be associated with surgical procedure steps. For example, biomarkers measured and monitored for the venous and arterial dissection step of a thoracic surgical procedure may include blood pressure, tissue perfusion pressure, edema, arteriosclerosis, collagen content, connective tissue thickness, etc., while biomarkers measured and monitored for the lymph node dissection step of a surgical procedure may include monitoring the patient's blood pressure. In one example, data regarding post-operative complications may be obtained from an EMR database in storage 20331, and data regarding staple or incision line leakage may be detected directly or inferred by the situational awareness system. Surgical procedure outcome data may be inferred by the situational awareness system from data received from various data sources, including the surgical device 20337, the sensing system 20336, and a database in storage 20331 to which the surgical hub 20326 is connected.

[0185] The surgical hub 20326 may transmit the measurement data and physical condition data received from the sensing system 20336 and / or data associated with the surgical devices 20337 to the analysis server 20338 for processing therein. Each of the analysis servers 20338 may include a memory and a processor coupled to the memory that may execute stored instructions to analyze the received data. The analysis servers 20338 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 20338 may determine optimal and / or suitable operating parameters for various types of modular devices, generate adjustments to control programs for the surgical devices 20337, and transmit (or “push”) updates or control programs to one or more surgical devices 20337. For example, the analysis system 20338 may correlate perioperative data received from the surgical hub 20336 with measurement data associated with the physiological state of the surgeon or HCP and / or the physiological state of the patient. The analysis system 20338 may determine when a surgical device 20337 should be controlled and send updates to the surgical hub 20326. The surgical hub 20326 may then transfer the control program to the associated surgical device 20337.

[0186] Further details regarding the computer-implemented wearable patient / surgeon wearable sensing system 20325, including the surgical hub 30326, one or more sensing systems 20336, and various surgical devices 20337 connectable thereto, are described in connection with Figures 5-7D.

[0187] FIG. 13 illustrates an example flow of a computing system, such as an audio augmented reality (AR) computing system, that adjusts AR content. In an example, the audio AR computing system may be or include earphones, a headset, headphones, etc., or a computing system that controls audio played through the earphones, a headset, headphones, etc. The audio AR computing system may receive audio data. The audio data may be or include one or more of audio data of measurement data associated with a user, ambient noise of an OR, audible feedback, audible information, etc. The audio AR computing system may generate AR content. For example, the audio AR computing system may generate AR content based on the received audio data. The AR content may be or include audible data, such as audible augmented feedback, from one or more computing devices and / or other computing systems. The audio AR computing system may obtain adjustment instructions (e.g., from other computing systems and / or the surgical computing system). The audio AR computing system may adjust the generated AR content based on the adjustment instructions.

[0188] At 29105, the audio AR computing system may receive audio data from one or more sensing systems and / or computing systems in an operating room (OR). The audio data may be or include audio data of measurement data associated with the user, ambient noise in the OR, audible feedback, audible information, etc. The audio data may be conditioned, filtered, and / or blocked. For example, the audio data may have filtered ambient noise in the OR.

[0189] In 29110, the audio AR computing system may generate AR content. For example, the audio AR computing system may generate AR content based on the received audio data. The generated AR content may be or include audible AR information. The audible AR can enhance what a user, such as a surgeon, hears. The audio AR computing system may allow the user to hear the generated AR, which may be or include audible AR information associated with the received audio data.

[0190] Generating AR content is further described in U.S. Patent Application No. 17 / 062,509, entitled "INTERACTIVE INFORMATION OVERLAY ON MULTIPLE SURGICAL DISPLAYS," filed October 2, 2020 (Attorney Docket No. END9287USNP16), which is incorporated herein by reference in its entirety.

[0191] At 29115, the audio AR computing system may obtain adjustment instructions. The adjustment instructions may indicate adjustment information for the generated AR content. In an example, the adjustment instructions may instruct to adjust the AR content by changing the voice of the AR content and blocking OR noise (e.g., ambient noise) that may be included in the AR content. In an example, the adjustment instructions may instruct to adjust audio AR settings associated with critical surgical steps and / or significant surgical steps. For example, the audio AR computing system may amplify and / or increase the volume of AR content (e.g., associated with critical surgical steps and / or significant surgical steps). The audio AR computing system may lower and / or decrease the volume of other AR content (e.g., non-critical surgical steps and / or non-critical surgical steps). The audio AR computing system may increase or decrease the frequency with which the generated AR content is sent to the user. In an example, if the audio AR computing system receives two or more audio data from other devices (e.g., other sensing systems and / or computing systems), the adjustment instructions may indicate the preferred audio data to send based on one or more of user preferences, priority information, and / or relevance to the current task and / or step of the surgery. For example, the AR adjustment instructions may be received from one or more computing systems in the OR (e.g., the surgical computing system and / or the surgical hub, etc.).

[0192] At 29115, the audio AR computing system may adjust the generated AR content. In an example, based on the adjustment instruction, the audio AR computing system may adjust the generated AR content. The audio AR computing system may change the voice of the AR content. The audio AR computing system may block out ambient noise in the OR. The audio AR computing system may amplify and / or increase the volume of the AR content, reduce and / or decrease the volume of the AR content, and / or increase or decrease the frequency of transmitting the generated AR content to the user. The audio AR computing system may select audio data from two or more audio data from the sensing system and / or computing system.

[0193] The audio AR computing system skips adjusting the generated AR content. For example, the audio AR computing system may allow the AR content (e.g., audible information) to pass through the AR content without filtering and / or adjusting. The audio AR computing system may determine that a surgical procedure is about to begin and / or non-critical tasks and / or steps of the surgical procedure (e.g., based on adjustment instructions). The audio AR computing system may allow the AR content to pass through (e.g., skip adjustments) and allow a user, such as a surgeon, to hear the ambient noise of the OR.

[0194] In an example, the audio AR computing system may adjust the generated AR content, for example, by canceling and / or blocking ambient noise and / or other audible data based on the adjustment instructions. For example, the audio AR computing system may determine that the next step (task) in a surgical procedure is a critical step. The audio AR computing system may cancel and / or block ambient noise and / or other audible data in the OR to provide a quiet environment for the user. A user of the audio AR computing system, such as a surgeon, can focus on the critical step. The audio AR computing system may enable the user to experience a quiet environment, such as reduced interaction with other HCPs in the OR and / or surrounding ORs. The audio AR computing system may eliminate distracting and / or overwhelming sounds (e.g., distracting sounds) from the AR content.

[0195] The audio AR computing system may adjust the AR content based on, for example, the adjustment instructions, and insert calming music and / or a voice that helps keep the user calm. For example, the audio AR computing system may adjust the AR content to provide white noise, calming music, and / or music preferred and / or pre-configured by the user. The audio AR computing system may transmit the adjusted AR content along with calming music or white noise to help the user focus on a current step associated with a surgical procedure.

[0196] In examples, the audio AR computing system may adjust the generated AR content by adjusting audio AR settings associated with the generated AR content. The audio AR computing system may adjust audio AR settings associated with the generated AR content. In examples, the audio AR computing system may amplify and / or increase the volume of the generated AR content. In examples, the audio AR computing system may reduce and / or decrease the volume of the generated AR content. In examples, the audio AR computing system may increase the frequency with which the generated AR content is transmitted to the user or decrease the frequency with which the generated AR content is transmitted to the user.

[0197] The audio AR computing system may adjust the AR content, such as the audio AR settings, based on the adjustment instructions. The adjustment instructions may be or include a surgical task instruction and / or a task importance instruction. The surgical task instruction may indicate a surgical task, such as a current surgical task being performed or a pending / next surgical task to be performed. The task importance instruction may indicate the importance and / or criticality of the surgical task.

[0198] In an example, the audio AR computing system may amplify and / or increase the volume of the AR content based on a surgical task instruction and / or a task importance instruction indicating that the surgical task is important and / or that the user can hear and focus on the amplified and / or increased volume of the AR content.

[0199] In examples, the audio AR computing system may reduce and / or decrease the volume of the AR content based on the surgical task instruction and / or task importance instruction indicating that the surgical task is not important (e.g., less important) and / or a non-critical task. The user may hear the reduced and / or decreased volume of the AR content and may relax.

[0200] The AR content may include multiple audio streams from multiple data sources. In an example, the audio AR computing system may identify the importance of the audio stream to a surgical step. The audio AR computing system may adjust the volume of the audio stream based on the current surgical step and the importance of the audio stream to the surgical step. For example, when the audio stream is important to the surgical step, the volume of the audio stream may be increased, and when the audio stream is not important to the surgical step, the volume of the audio stream may be decreased.

[0201] In examples, the AR content may include multiple audio streams from multiple data sources. For example, the AR content may be or may include multiple audio data from multiple sensing systems. The audio AR computing system may receive audio data from sensing systems in the OR. The audio AR computing system may receive adjustment instructions indicating user preference settings associated with the surgical procedure. For example, the user preference settings may be or may include preferred measurement data for the surgical procedure. The audio AR computing system may receive other audio data from other sensing systems in the OR. The audio AR computing system may select preferred audio data. For example, the audio AR computing system may select preferred audio data from multiple audio data from multiple sensing systems indicated in the user preference settings. As described herein, the audio AR computing system may adjust the AR content by reducing (e.g., decreasing) the volume of unselected audio data from the sensing systems and / or amplifying (e.g., increasing) the volume of selected audio data from the sensing systems. The audio AR computing system may adjust the AR content by increasing the frequency of selected AR content and / or decreasing the frequency of unselected AR content. The audio AR computing system may adjust the AR content by blocking out unselected audio data.

[0202] In an example, the audio AR computing system may increase the frequency at which generated AR content is played for a user based on a surgical task instruction and / or a task importance instruction. The task importance instruction may indicate that the surgical task is an important (e.g., critical) task. In an example, the audio AR computing system may increase the frequency of generated AR content when an emergency occurs (e.g., when the patient's measurement data falls below or exceeds a threshold level). For example, when measurement data associated with a patient's heart rate suddenly changes, the audio AR computing system may increase the frequency at which the heart rate measurement data is notified to a user, such as a surgeon. The user can hear the heart rate measurement data at an increased frequency and be aware of the real-time measurement data.

[0203] In an example, the audio AR computing system may reduce the frequency with which AR content is sent to a user based on a surgical task instruction and / or a task importance instruction indicating that the surgical task is an important (e.g., critical) task. In an example, the audio AR computing system may reduce the frequency with which AR content is sent to a user if the emergency has passed (e.g., if the patient's measurement data has returned to normal levels). For example, if the measurement data is associated with the patient's heart rate and the heart rate data has returned to normal (e.g., if the emergency has been averted), the audio AR computing system may reduce the frequency with which the audio AR computing system notifies the user of the heart rate measurement data. Once the emergency has been averted and / or the patient has stabilized, a user, such as a surgeon, may be able to listen to and / or focus on other AR content.

[0204] In an example, the audio AR computing system may adjust AR content, such as a voice associated with the AR content. For example, the adjustment instructions may be or include user preferences and / or user settings. Based on the user preferences and / or user settings, the audio AR computing system may change the voice endings of the AR content to a different voice. For example, the different voice may be or include Morgan Freeman, Denzel Washington, Darth Vader, and / or other voices that the user prefers to hear.

[0205] In an example, the audio AR computing system may adjust the AR content by translating the language. For example, the adjustment instruction may indicate that the AR content is in a different language, e.g., non-English. The audio AR computing system may translate the AR content in real time into English or a language the user can understand, facilitating better communication. In an example, a user, such as a surgeon, may travel to a foreign country and / or work with HCPs who are not fluent in the language the user speaks. For example, the surgeon may travel to another location (e.g., based on a specialty and / or an international program, such as Doctors Without Borders). The surgeon may not be fluent in the local language and may not be able to understand what is being said in the OR and / or surrounding conversations. The audio AR computing system may adjust the AR content and / or audio data associated with the OR's audio data in another language and may adjust the AR content by translating the AR content, e.g., in real time, into a language the user can understand.

[0206] In an example, the audio AR computing system can adjust the AR content based on a sound source location indication in the adjustment instructions. For example, the sound source location indication may indicate a sound source location of audible data associated with the AR content. The audio AR computing system may adjust the AR content based on the sound source location indication. In an example, if the sound source location indication indicates that the audible data is originating from outside the target area (e.g., outside the OR), the audio AR computing system may adjust the AR content by canceling the audible data originating from outside the OR. In an example, the audio AR computing system can anticipate audible data originating from outside the OR. For example, a user of the audio AR computing system can anticipate a phone call from an organ transplant specialist, another surgeon, an HCP in a different OR, a specialist in a different location (a different country), a technician from a surgical instrument company, etc. If the audio AR computing system determines that the sound source location associated with the audio data is an expected source location, it may adjust the AR content by allowing the audio data originating from outside the OR.

[0207] In examples, the audio AR computing system may adjust the AR content by selecting and / or prioritizing AR content, for example, based on adjustment instructions. For example, the AR content may be or include audible information from one or more computing devices and / or computing systems. The adjustment instructions may be or include prioritization instructions indicating a priority of the audible data. If the audio AR computing system determines that the generated AR content is or includes two or more pieces of audible data and / or audible information, the audio AR computing system may adjust the AR content by selecting the audible data and / or audible information based on the indicated prioritization information. The audio AR computing system may amplify and / or increase the volume of the prioritized audible data. In examples, the audio AR computing system may cancel other audible data. In examples, the audio AR computing system may reduce and / or decrease the volume of non-prioritized audible data. The priority of the audio data / audible information may be set via a user preference instruction indicating a preference for the audible data.

[0208] In an example, the audio AR computing system may receive an ambient noise level indication. The ambient noise level indication may indicate the ambient noise level in the operating room. In an example, the audio AR computing system may determine the ambient noise level in the OR. If the audio AR computing system determines that the ambient noise level is below a threshold ambient noise level, the audio AR computing system may determine that a critical surgical task should be performed or is being performed. In an example, the audio AR computing system may adjust the AR content based on the ambient noise level being below the threshold ambient noise level. For example, the audio AR computing system may cancel certain audible data in the AR content. For example, the audio AR computing system may provide a quiet and / or calming environment for the user to concentrate. In an example, the audio AR computing system may send a critical task indication to the surgical computing system. For example, the critical task indication may indicate that a critical surgical task should be performed (or is being performed) in the OR. The computing system may send one or more alerts to other HCPs in the OR that an upcoming task involves a critical surgical task.

[0209] In an example, the audio AR computing system may send a user input request to the surgical computing system. The audio AR computing system may send the user input request to the surgical computing system before adjusting the AR content. If the user does not provide input to the user input request (e.g., confirmation of the proposed AR adjustment), the audio AR computing system may skip adjusting the AR content. In an example, the audio AR computing system may determine that user input has not been registered for a certain period of time (e.g., a preconfigured time), and the audio AR computing system may send a reminder to the user regarding the user input request. In an example, if the audio AR computing system determines that user input has not been registered for a certain period of time, the audio AR computing system may refer to preconfigured settings (e.g., default settings). The user may have preselected and / or preconfigured default settings (e.g., the volume level of the AR content and / or the frequency of receiving the AR content). For example, the user may have preconfigured the audio AR computing system to adjust the AR content to the default settings if the audio AR computing system does not register user input after a preconfigured time (e.g., after 20 seconds) and / or after the audio AR computing system has sent a reminder.

[0210] The audio AR computing system may receive user input associated with (e.g., in response to) a user input request. If the audio AR computing system receives the user input, the audio AR computing system may adjust the AR content (e.g., further adjust the AR content) based on the user input. For example, as described herein, based on the user input, the audio AR computing system may further adjust the AR content by amplifying (e.g., increasing) the volume of the AR content, reducing (e.g., decreasing) the volume of the AR content, increasing the frequency of the AR content, and / or decreasing the frequency of the AR content.

[0211] The audio AR computing system may adjust the generated AR content based on adjustment instructions, which may be or include, for example, surgical step instructions. For example, the surgical step instructions may indicate the current and / or next surgical steps associated with the surgical procedure. The audio AR computing system may detect and / or recognize the current and / or next surgical steps, for example, based on the surgical step instructions. In an example, the audio AR computing system may adjust the AR content based on a determination (e.g., situational awareness) that the audio data of an HCP role is relevant to the current and / or next surgical step. As described herein, the audio AR computing system may adjust the AR content by allowing the audio data of the relevant HCP role (e.g., head nurse) and canceling audio data associated with other HCP roles (e.g., and / or ambient noise).

[0212] Determining user or HCP roles is further described in co-pending Attorney Docket No. END9290USNP17, entitled "ACTIVE RECOGNITION AND PAIRING SENSING SYSTEMS," which is incorporated herein by reference in its entirety. For example, as described herein, the AR computing system may receive user role identification data from one or more sensing systems in the OR. The user role identification data may be or include information for identifying a user role. The surgical computing system may identify a user role for the user in the OR based on the received user role identification data. The user role for the user in the OR may be or include at least one of a surgeon, a nurse, a patient, a hospital staff member, or an HCP.

[0213] In an example, the audio AR computing system may receive audio data associated with an HCP role in the OR (e.g., a resident). The audio AR computing system may receive other audio data associated with another HCP role in the OR (e.g., a head nurse). The audio AR computing system may determine whether the audio data is relevant to the surgical step indicated in the surgical step instructions. For example, the audio AR computing system may determine whether audio data of a resident and / or a head nurse in the OR is relevant to the surgical step indicated in the surgical step instructions. If the audio AR computing system determines that the audio data is relevant to the surgical step, the audio AR computing system may adjust the AR content by allowing the relevant audio data. For example, if the audio AR computing system determines that the resident's audio data is relevant to the surgical step and the head nurse's audio data is unrelated to the surgical step, the audio AR computing system may adjust the AR content by passing (e.g., allowing) the resident's audio data and blocking the head nurse's audio data.

[0214] Determining user or HCP roles is further described in co-pending Attorney Docket No. END9290USNP17, entitled "ACTIVE RECOGNITION AND PAIRING SENSING SYSTEMS," which is incorporated herein by reference in its entirety.

[0215] For example, the AR computing system may receive user role identification data from one or more sensing systems in the OR. The user role identification data may be or include information for identifying a user role. The surgical computing system may identify a user role for a user in the OR based on the received user role identification data. The user role of the user in the OR may be or include at least one of a surgeon, a nurse, a patient, a hospital staff member, or an HCP. Based on the identified user role, the surgical computing system may generate surgical assistance information for the user in the OR. The surgical assistance information may be or include information associated with a surgical procedure related to the identified user role. The AR computing system may transmit relevant information to the identified user, for example, via AR content, as described herein.

[0216] The user role identification data may be or may include one or more of the following: a user's proximity to surgical instruments, a user's location and / or location tracking information within the OR, an interaction between the user and at least one HCP, one or more surgical procedure activities, or visual data of the user within the OR. For example, a sensing system may be worn by a user, such as a surgeon. The sensing system may monitor and / or store information regarding the sensing system's proximity to surgical instruments. The sensing system may store position tracking information of the surgeon during a surgical procedure. The sensing system may detect and / or store the surgeon's surgical procedure activities. The sensing system may transmit such user role identification data to the surgical computing system.

[0217] For example, as described herein, the AR computing system may generate AR content for a user based on an identified user role. Different AR content may be generated for different users based on the respective user roles identified via the sensing system. The AR content may be or include instructions on how to use a surgical instrument and / or an operating manual for the surgical instrument associated with the identified user role. The surgical computing system may transmit the generated AR content to the identified user. For example, the surgical computing system may transmit the AR content to an AR device associated with the user.

[0218] In examples, the audio AR computing system may adjust the AR content by amplifying (e.g., increasing) the volume of audio data associated with HCP roles related to surgical steps, reducing (e.g., decreasing) the volume of audio data associated with HCP roles unrelated to surgical steps, increasing the frequency of audio data associated with HCP roles related to surgical steps, and / or decreasing the frequency of audio data associated with HCP roles unrelated to surgical steps.

[0219] As described herein, the audio AR computing system may adjust the AR content based on the OR's recognition, for example, based on an ambient noise level indication. If the audio AR computing system determines that the ambient noise level indication is below a threshold noise level, the audio AR computing system may determine that the current and / or next surgical step (e.g., task) is a critical step (e.g., task). The audio AR computing system may adjust the AR content by allowing relevant audible data associated with critical surgical tasks and / or blocking other audible data associated with non-critical surgical tasks (e.g., ambient noise). The audio AR computing system may adjust the AR content by amplifying (e.g., increasing) the volume of audio data associated with critical surgical tasks, increasing the frequency of audio data associated with critical surgical tasks, reducing (e.g., decreasing) the volume of audio data associated with critical surgical tasks, and / or decreasing the frequency of audio data associated with critical surgical tasks.

[0220] The audio AR computing system may adjust the AR content-based recognition of the user's state. For example, the audio AR computing system may determine that the user's stress level has increased, e.g., based on measurement data associated with the user. Based on the increased stress level, the audio AR computing system may derive that the current and / or next surgical tasks are critical tasks. If the audio AR computing system determines that the current and / or next surgical tasks are critical tasks, the audio AR computing system may adjust the AR content. For example, as described herein, the audio AR computing system may adjust the AR content by passing relevant audible data associated with critical surgical tasks and / or blocking other audible data (e.g., ambient noise) associated with non-critical surgical tasks. Upon detecting the user's increased stress level, the audio AR computing system may adjust the AR content by amplifying (e.g., increasing) the volume of audio data associated with critical surgical tasks, increasing the frequency of audio data associated with critical surgical tasks, reducing (e.g., decreasing) the volume of audio data associated with critical surgical tasks, and / or decreasing the frequency of audio data associated with critical surgical tasks. Upon detecting an elevated stress level of the user, the audio AR computing system may adjust the AR content by inserting calming audio.

[0221] Determining stress levels is further described in co-pending Attorney Docket No. END9290USNP2 entitled "ADAPTABLE SURGICAL INSTRUMENT CONTROL," which is incorporated herein by reference in its entirety.

[0222] For example, the computing system may receive measurement data from one of the sensing systems associated with a user in the operating room (e.g., a sensing system associated with a surgeon). The computing system may also receive measurement data from one of the sensing systems associated with a user in the operating room that indicates a higher stress level of the user. For example, a higher stress level may be indicated by a change in the user's heart rate from a baseline value. The computing system may derive this inference by cross-referencing receipt of data from the corresponding sensing systems. The computing system may transmit surgical assistance information to the identified user as described herein.

[0223] The audio AR computing system may adjust the AR content based on a recognition of a user state. For example, the audio AR computing system may determine that a user (e.g., a user wearing the audio AR computing system) has an elevated fatigue level, e.g., based on measurement data associated with the user. Based on the elevated fatigue level, the audio AR computing system may recognize and / or determine that the user may need to concentrate. If the audio AR computing system determines that the user needs to concentrate, the audio AR computing system may adjust the AR content. For example, as described herein, the audio AR computing system may adjust the AR content by allowing relevant audible data associated with the current surgical task and / or blocking other audible data not associated with the current surgical task (e.g., ambient noise). The audio AR computing system may adjust the AR content by one or more of amplifying (e.g., increasing) the volume of audio data associated with the current surgical task and / or increasing the frequency of audio data associated with the current surgical task. The audible AR component may adjust the AR content by one or more of reducing (e.g., decreasing) the volume of audio data not associated with the current surgical task and / or decreasing the frequency of audio data not associated with the current surgical task.

[0224] Determining fatigue levels is further described in co-pending Attorney Docket No. END9290USNP2 entitled "ADAPTABLE SURGICAL INSTRUMENT CONTROL," which is incorporated herein by reference in its entirety.

[0225] For example, the AR computing system may receive measurement data from one of the sensing systems associated with users in the OR (e.g., a sensing system associated with a surgeon). The measurement data may indicate that a user, such as a surgeon, is making excessive input changes, which may be referred to as overcorrection, in response to a perceived error. The AR computing system may interpret the repeated corrections, overcorrection, or oscillatory responses as indicators of fatigue and / or elevated fatigue levels associated with the identified user.

[0226] The AR computing system may be configured to analyze the usage data and / or measurement data to determine whether a user working in the OR is experiencing fatigue, and if so, to modify the operation of the surgical instrument and / or provide a notification associated with the fatigue level. For example, the AR computing system may monitor user input to the surgical instrument (e.g., from the surgical instrument and / or from a sensing system). The user input to the surgical instrument may include input that results in trembling of the surgical instrument. The trembling, whether intentional or unintentional, may be detected by one or more sensing systems (e.g., acceleration sensors) that provide data regarding the movement and orientation of the surgical instrument. The detected data may indicate the magnitude and frequency of any tremor. The surgical instrument may generate usage data associated with the monitored user input. The usage data may indicate the input to the surgical instrument, including, for example, movement of all or a portion of the surgical instrument, including trembling. The usage data may be communicated to the AR computing system.

[0227] Data may be collected from a sensing system that may be attached to the user of the surgical instrument as well as other HCPs who may assist in the OR. Accelerometers may be attached to the user's hands, wrists, and / or arms. Accelerometers may also be attached to the user's torso to collect data associated with body movements, including sway and body tremors. The accelerometers may generate data regarding the movement and orientation of the user's hands and / or arms. The data may include the magnitude and frequency of movements, including sway. The sensing system (which may be or include, for example, an accelerometer) may collect biomarker data from the user, including data associated with heart rate, respiration, body temperature, etc. The sensing system may collect data associated with the hydration / dehydration of the corresponding user operating the surgical instrument as well as other users assisting in the OR. The collected data may be communicated to the AR computing system.

[0228] The AR computing system may receive usage data from surgical instruments and may receive sensor data from a sensing system corresponding to a user in the OR. The AR computing system may identify and / or store the received data in association with timestamp data indicating the time the data corresponding to the user was collected.

[0229] The AR computing system may determine a fatigue level of a user operating a surgical instrument and assisting in an OR based on the received use data and / or sensor data. The AR computing system may determine a duration associated with a surgical procedure based on the received use data and / or sensor data. For each user, the AR computing system may determine values ​​associated with time in the OR, time spent standing in the OR, and time spent physically working. The AR computing system may determine a user's fatigue level based on time spent performing surgery.

[0230] The AR computing system may determine physical indicators of fatigue based on the received usage data and / or sensor data. The AR computing system may determine that the user is fatigued if the received data indicates that the user is shaking or unsteady. The AR computing system may determine that the user is fatigued if the received data indicates that the user is exhibiting a tremor.

[0231] The AR computing system may determine a value associated with hydration / dehydration of users in the OR based on the received usage data and / or sensor data. Dehydration can affect energy levels and make a person feel tired and fatigued. Low body fluids tend to increase heart rate. The AR computing system may analyze heart rate data in the context of hydration levels and distinguish stress and other heart rate-increasing events from hydration. The AR computing system may employ baseline measures to distinguish acute events from ongoing chronic events and to distinguish between fatigue and dehydration associated with each user in the OR.

[0232] The AR computing system may calculate a weighted measure of fatigue for the user operating the surgical instrument as well as other users in the OR. The weighted measure of fatigue may be based on cumulative collaborative events and contributions. For example, the weighted measure of fatigue may be based on the intensity of stress experienced by the user and the force exerted by the user over time when controlling an actuator such as a closure trigger over time.

[0233] If the AR computing system determines that the user is experiencing fatigue, the AR computing system may decide to communicate a control function to the surgical instrument to other AR computing systems associated with the HCP in the OR and / or the AR computing system of the user experiencing elevated fatigue levels. The communicated control function may be or include fatigue control or adjustments and modulations to compensate for fatigue. A control function for implementing fatigue control may indicate reducing the force required to perform an action. For example, the control function may indicate reducing the force required to apply to a closure trigger to actuate the clamp jaws of a surgical instrument. The control function may indicate increasing the sensitivity of the closure trigger. The control function may indicate increasing the delay or latency in responding to user input. The control function may indicate slowing actuation to provide additional time before action.

[0234] If the computing system determines that the user is experiencing fatigue, the AR computing system may also determine to communicate with a control function to provide a notification regarding the fatigue. The AR computing system may determine that a notification regarding the fatigue may be provided to the user by a surgical instrument. The AR computing system may determine that the notification may provide more usage steps to the operator. The AR computing system may also determine that a notification regarding the fatigue level may be provided to someone in the OR other than the HCP responsible for the instrument. Such a notification may be displayed on a display system in or near the OR.

[0235] The AR computing system may communicate instructions for control functions associated with fatigue control, which may be communicated to the surgical instrument, the AR computing system, and / or to other systems within the OR, such as displays that may be employed to provide notifications.

[0236] The surgical instrument and display can receive and provide notification of the control function indication to implement fatigue control. The surgical instrument may decide to operate in accordance with the fatigue control indication. The instrument may reduce the force required to activate and / or operate a closure trigger. The surgical instrument may increase the delay or latency between requesting an action, e.g., applying force to a closure trigger, and performing the corresponding action, e.g., closing the jaws. The surgical instrument can slow actuation in response to an input, thereby providing more time for the operator to position the surgical instrument.

[0237] If the control function indicates that notifications are provided, the surgical instrument may provide physical tactile feedback as well as visual feedback. The display may also provide visual feedback regarding fatigue. The notifications may provide usage steps to minimize overlooking details.

[0238] In an example, if the audio AR computing system receives critical audible information associated with a patient (e.g., a sudden change in the patient's condition), the audio AR computing system may enable the critical audible information associated with the patient's condition to be transmitted. For example, the audio AR computing system may exclude the critical audible information associated with the patient's condition from adjustment (e.g., cancellation).

[0239] A computing system, such as an audio AR computing system and / or a visual AR computing system, may interpolate data, such as AR data and / or AR content, to overlay with the augmented array. In an example, the audio AR computing system may interpolate audible AR content, and a user of the audio AR computing system may hear overall changes to the AR content (e.g., patient measurement data associated with the patient's condition). In an example, the visual AR computing system may interpolate visual AR content, and a user of the visual AR computing system may see overall changes to the AR content (e.g., patient measurement data associated with the patient's condition).

[0240] In an example, the audio AR computing system may provide audible information associated with the gradient of a marker on a patient (e.g., the patient's body) or the gradient over time to the patient's improved condition. A user of the audio AR computing system may understand the patient's condition through the audio AR computing system and / or the audible AR content.

[0241] In an example, the visual AR computing system may provide visual information associated with the gradient of a marker on a patient (e.g., the patient's body) or the gradient over time of the patient's improved condition. A user of the visual AR computing system may understand the patient's condition through the visual AR computing system and / or the visual AR content.

[0242] The audible and / or visual AR content may be or include one or more pieces of information from a camera in the OR, an image of the patient's body (e.g., MRI, MRA, etc.), a camera inside the patient's body, etc.

[0243] The audible and / or visual AR content may be or include one or more information associated with a patient's temperature, such as the patient's core temperature and / or the patient's peripheral temperature. The audible and / or visual AR content may be or include a temperature gradient plotted on the patient's body. An audio AR computing system may provide the audible AR content, and a user, such as a surgeon, may hear the audible AR content (e.g., the patient's temperature information and / or the patient's gradient temperature information). A visual AR computing system may provide visual AR content, such as overlay gradient temperature information for the patient. A user of the visual AR computing system may view the visual AR content and / or monitor the fluctuating patterns of the patient's temperature.

[0244] In examples, an AR computing system (e.g., an audio AR computing system and / or a visual AR computing system) may receive AR content, such as patient measurement data, from one or more other computing systems and / or computing devices. The AR computing system may receive and / or collect the patient measurement data and generate AR content associated with the patient measurement data. The AR computing system may transmit audible information associated with the patient measurement data. The AR computing system may present visual information associated with the patient measurement data. The AR content may be or include patient gradient information and / or changes in the patient measurement data over time.

[0245] An AR computing system (e.g., an audio AR computing system and / or a visual AR computing system) may provide AR content that may be or include measurement data. In an example, an audio AR computing system may provide audible AR content about a patient's measurement data, e.g., locally to a user wearing the audio AR computing system. In an example, a visual AR computing system may provide visual AR content about a patient's measurement data, e.g., locally to a user wearing the visual AR computing system. The AR content may be an information overlay of the patient's measurement data. The AR content may provide data depth to a user, e.g., via an AR overlay.

[0246] The AR computing system may receive one or more measurement data from one or more sensing systems. For example, the AR computing system may receive one or more measurement data from one or more sensing systems located in an OR. The measurement data may be or include measurement data of a user, such as a patient and / or a surgeon.

[0247] In an example, the audio AR computing system may generate audible AR content based on the measurement data. As described herein, the audio AR computing system can overlay audible information on the measurement data and generate and / or adjust the AR content. For example, the audio AR computing system may overlay audible AR content associated with monitoring and / or real-time measurement data of a patient and / or user (e.g., a surgeon, etc.). In an example, the audio AR computing system can transmit the AR content using an audio output associated with the audio AR computing system to provide the AR content locally to the user. In an example, the audio AR computing system can share the audible AR content (e.g., for broadcasting) to speakers and / or audio outputs connected to the OR and / or to other audio AR computing systems associated with other HCPs in the OR.

[0248] In examples, the visual AR computing system may generate visual AR content based on the measurement data. As described herein, the visual AR computing system may overlay visual information on one or more measurement data and generate and / or adjust the AR content. For example, the visual AR computing system may overlay visual AR content associated with monitoring and / or real-time measurement data of a patient and / or user (e.g., a surgeon, etc.). In examples, the visual AR computing system may transmit the AR content using a display associated with the visual AR computing system to provide the AR content locally to the user. In examples, the visual AR computing system may share (e.g., for broadcasting) the visual AR content with displays and / or monitors connected to the OR and / or other visual AR computing systems of other users (e.g., HCPs) in the OR.

[0249] In an example, as described herein, the audio AR computing system and / or the visual AR computing system may request user input before sharing information. If the audio AR computing system and / or the visual AR computing system does not receive user input, the audio AR computing system and / or the visual AR computing system may send a reminder user input request and / or follow preconfigured settings (e.g., default settings). In an example, the preconfigured settings may be or may include skipping sharing information to other HCPs and / or skipping broadcasting to ORs. In an example, the preconfigured settings may be or may include sharing measurement data to HCPs and / or broadcasting to ORs.

[0250] In an example, if the audio and / or visual AR computing system does not receive user input (e.g., from a primary user such as a surgeon), the audio and / or visual AR computing system may send a user input request to one or more other HCPs in the operating room. For example, if the audio and / or visual AR computing system does not receive input from the surgeon, the audio and / or visual AR computing system may send a user input request to the head nurse. The head nurse may provide user input. The surgeon may handle a list of HCPs, such as the head nurse. The head nurse may remember the surgeon's preferences and / or previous orders from the surgeon. Based on the surgeon's orders and / or known preferences, another HCP, such as the head nurse, may provide user input for the surgeon. The audio and / or visual AR computing system may share the AR content locally and / or broadcast the AR content within the OR and / or to other HCPs within the OR.

[0251] In examples, the audio AR computing system and / or the visual AR computing system may receive measurement data from one or more sensing systems and / or surgical computing systems. As described herein, the audio AR computing system and / or the visual AR computing system may adjust AR content (e.g., audible AR content or visual AR content). The AR device may display the received data (e.g., wearable data). In examples, the audio AR computing system and / or the visual AR computing system may highlight particular measurement data (e.g., important information such as a patient's blood pressure and / or heart monitor information).

[0252] In examples, the audio AR computing system can adjust the AR content and amplify (e.g., increase the volume of) audible information associated with particular measurement data relevant to and / or important to the current surgical procedure. The audio AR computing system may adjust the AR based on perceptions of the surgical procedure, the atmosphere of the OR, and / or interactions between HCPs, as described herein. The audio AR computing system may adjust the AR content based on adjustment instructions received from the surgical computing system indicating important and / or relevant measurement data associated with the current surgical task.

[0253] In examples, the visual AR computing system can adjust the AR content to increase the resolution of visual information associated with particular measurement data relevant and / or important to the current surgical procedure. For example, the visual AR computing system may provide greater resolution of measurement data relevant and / or important to the current surgical procedure. The visual AR computing system may adjust the AR based on perceptions of the surgical procedure, the atmosphere of the OR, and / or interactions between HCPs, as described herein. The visual AR computing system may adjust the AR content based on adjustment instructions received from the surgical computing system indicating important and / or relevant measurement data associated with the current surgical task.

[0254] The audio AR computing system and / or the visual AR computing system can provide the measurement data simultaneously and / or sequentially. For example, the audio AR computing system can adjust AR content, which can be or include the measurement data, to continuously provide audible information associated with the measurement data. In an example, the audio AR computing system can provide the audible information in the same tone. In an example, the audio AR computing system can adjust the AR content to emphasize important and / or relevant measurement data with different tones, volumes, and / or voices.

[0255] The visual AR computing system can adjust the AR content to simultaneously display one or more (e.g., all) measurement data from the sensing system. In an example, the visual AR computing system can display the measurement data at the same resolution. In an example, the visual AR computing system can adjust the AR content to provide higher and / or different resolutions to highlight important and / or relevant measurement data. In an example, the visual AR computing system can adjust the AR content to enlarge important and / or relevant measurement data and / or omit other measurement data.

[0256] In examples, the AR computing system may select audible and / or visual information to tailor the AR content based on the current step of the surgery. For example, as described herein, the AR computing system may be contextually aware of the current step and / or task of the surgery. The AR computing system may select relevant AR information for tailoring the AR content. The AR may transmit unselected audible and / or visual AR information to the HCP. For example, a surgeon may receive AR content with relevant AR information associated with the current step of the surgical procedure. In examples, other HCPs in the OR may receive the same information. In examples, other HCPs in the OR may receive other AR information or monitor the information.

[0257] The AR computing system may send requests to one or more sensing systems and / or surgical computing systems. The requests may be or include additional measurement data and / or updated data. For example, based on the current step and / or task, the AR computing system may prioritize AR content and / or skip receiving AR information updates. In an example, during an emergency, the AR computing system may skip receiving the patient's heart rate trace information, electrocardiogram (EKG) information, and / or heart rate variability information. After the emergency, the AR computing system may resume receiving such information. For example, the AR system may send an update request for the skipped measurement data to one or more sensing systems and / or surgical computing systems. The AR computing system may receive the updated and / or monitored measurement data.

[0258] In an example, the AR computing system may receive one or more measurement data from one or more corresponding sensing systems associated with the patient. The sensing systems may be tracking the measurement data associated with the patient. For example, the measurement data may be or include the patient's heart rate trace information, EKG information, and / or heart rate variability information. The sensing systems may have the patient's measurement data over a period of time (e.g., before surgery) and may provide a history of the measurement data. The measurement data may be or include the patient's real-time measurement data.

[0259] A user of the AR computing system can pre-configure (e.g., pre-set) AR settings associated with receiving audible and / or visual AR information within the AR content. For example, the user may configure the frequency with which the audible and / or visual AR information is received (e.g., every 5 seconds or every minute). The user may configure the volume and / or resolution of the audible and / or visual AR information. The user may configure the AR settings associated with the audible and / or visual AR information before and / or during surgery based on historical user preference data.

[0260] Measurement data from the sensing system may be used for risk assessment and may be applied to a surgical procedure (e.g., a suitable surgical procedure). A computing system may use the data to assess the risk of a surgical procedure. A computing system may use the data to inform a go / no-go surgical decision.

[0261] The sensing system may collect measurement data. For example, the sensing system may collect measurement data (e.g., sensor data) before a surgical procedure. The sensing system may monitor one or more variables (e.g., specific variables) and provide frequency updates to an HCP, such as a surgeon, before surgery. The measurement data may help notify the surgeon whether acceptable conditions are in place before a scheduled surgery. In an example, the international normalized ratio (INR) may be a metric used to assess blood clotting, for example, in patients receiving Coumadin (e.g., a fairly common anticoagulant). Elevated levels (e.g., elevated levels of measurement data) may be common for patients receiving anticoagulation therapy. Elevated levels may be associated with bleeding complications in elective and / or emergency surgical procedures.

[0262] The computing system can monitor absolute INR values ​​and / or changes (e.g., trends) in INR values ​​prior to surgery. Absolute INR values, INR values, and / or trends in INR values ​​can indicate a patient's readiness for surgery. Guidelines (e.g., surgical guidelines) may recommend discontinuing Coumadin for 5-6 days prior to surgery and / or administering reversal therapy approximately 6 hours prior to surgery. If the patient is in the hospital, the patient's vitals and / or other information can be easily tracked. If the patient is not in the hospital, the patient's vitals and / or other information cannot be easily tracked. If the patient is not in the hospital, the patient's vitals and / or other information may be tracked the day before or the day of surgery. Having patient vitals and / or other information the day before or the day of surgery can lead to planning challenges and / or increased bleeding risk in the OR (e.g., and / or increased procedure costs).

[0263] One or more computing systems and / or one or more sensing systems can communicate and share measurement data (e.g., laboratory tests) and provide an overall analysis (e.g., an improved overall analysis). The computing devices can interact with other data sources (e.g., hubs and / or sensing systems), which may impact the pre- or post-operative care of the patient.

[0264] Combining multiple data sources may provide patient care instructions (e.g., optimal patient care instructions). For example, preoperative care changes of dietary changes and / or renal function medication changes may be implemented. For example, preoperative care (e.g., specific dietary changes and / or risk stratification for dialysis) may be implemented. In an example, low serum albumin levels may be associated with poor surgical outcomes (e.g., increased morbidity and / or mortality). Low serum albumin levels may or may not be nutrition-related. In an example, combining serum albumin measurements with changes in weight (e.g., measurement data from a wireless scale) may help control low albumin from malnutrition, low albumin from kidney disease, and / or other conditions. In an example, bioimpedance analysis may be combined with measurement data from a scale. The combined bioimpedance analysis and measurement data from the scale may help identify water changes in a patient's weight.

[0265] The computing system may monitor a patient's preconditioning. For example, the computing system may monitor and / or seek a readiness threshold from the patient's monitored preconditioning. The patient's preconditioning may prepare the patient for surgery and / or may monitor the patient to achieve a threshold set by an HCP, such as a surgeon.

[0266] The computing system may use preoperative patient monitoring data and / or adjustments to acclimate the body based on the duration of surgery. The sensing system may collect measurement data. For example, the measurement data may include heart rate, respiratory rate, body temperature, sleep, mental state, etc. The computing system may determine when the patient should undergo surgery based on the measurement data.

[0267] In an example (e.g., in addition to and / or instead of the example), the computing system may use measurements to train the body and / or subconscious mind to adjust over a period of time. Based on the monitored data, the computing system may set one or more triggers at specific times to lower anxiety, reduce heart rate and breathing rate to minimize inflammation, provide instructions to the user to rest, etc. The computing system may layer triggers over time. For example, the computing system can layer triggers over time, so that the body and mind adjust and become more relaxed at the time of surgery.

[0268] In examples (e.g., in addition to and / or instead of examples), the computing system may utilize measurement data and / or may be based on one or more triggers. The one or more triggers may pop up a video on the patient's device, such as a phone, for viewing. The patient may watch the video to relax, slow their pulse rate, and / or alter their breathing. The patient may listen to an audio recording, for example, to intentionally alter the frequency of the patient's brain waves. For example, the patient's brain waves may be categorized into specific frequencies depending on what the patient is doing at a given time. If the patient is engaged in a specific motor function, the brain waves may be gamma. If the patient is fully conscious and / or actively focused, the brain waves may be beta. If the patient is relaxed, the brain waves may be alpha. If the patient is drowsy and / or in light sleep, the brain waves may be theta. If the patient is deep asleep, the brain waves may be delta.

[0269] Binaural beats can occur when two tones are played at different frequencies. Binaural beats can trigger the patient's brain waves to follow different patterns. For example, if a computing device (e.g., using measurement data) wants to transition a patient's state from a stressed state to a relaxed state, the computing device can play a sound, and the patient can hear the sound that triggers an alpha state.

[0270] Audio programs can help reprogram the patient's subconscious mind, for example, by creating a more receptive forum for installing positive messages. The subconscious mind can be more receptive to new information when the patient / body is relaxed, such as in an alpha or theta state.

[0271] Using brain-entrainment audio programs and / or affirmations or visualizations can be a powerful combination: the patient's subconscious mind can lower its defenses and easily absorb messages that the HCP and / or computing device may wish to program.

[0272] When two or more HCPs are involved and the HCPs are one or more sites of data communication between the HCPs, coordination and / or treatment between the HCPs may be coordinated (eg, improved).

[0273] The computing device (e.g., a wearable device) may provide reminder(s) to the user. For example, the computing device may provide the user with a reminder of information provided by the HCP, such as upon discharge from the hospital. If the reminders are not helpful in addressing distraction, the computing device may link to cellular, WIFI, and / or other networks to allow the HCP to interact with the user in real time. The reminders may serve as reminders and a means for the user to address what they need to do to improve compliance and / or recovery. In examples, the reminders may be or include exercise(s) to be performed daily and / or medication(s) to be taken at specific times. The computing device may remind the user and / or detect that the user is engaging in recommended exercise and / or taking medication. If the computing device does not detect an event and / or the underlying measurement data (e.g., biomarkers) indicate a lack of improvement, the computing system can be used to understand whether the user is performing exercises correctly and / or taking medications on time. The computing system can notify the HCP, and the computing system can help remind the user to perform the activity.

[0274] After surgery, the surgeon may provide information regarding movement restrictions and / or required exercise to the primary care physician, physical therapist, and / or other HCP. Other HCPs may modify medications the user is taking (e.g., temporarily or in response to measured parameters). Other HCPs may monitor and / or ensure compliance with pre- and / or post-operative regimens such as meals, rest, etc. Other HCPs may have one or more measurement data (e.g., biomarker) thresholds that now hold higher importance and / or may trigger intervention if the measurement data (e.g., biomarkers) do not change over time as expected.

[0275] One or more supporting HCPs may record the user's progress and / or compliance. The surgeon may have the progress and / or compliance available when the surgeon reviews the patient's recovery.

[0276] The computing system may include an antenna (e.g., a flexible antenna) and may separate the detection system and the communication system. In an example, the computing system may use signal strength, noise, and / or directional antennas to selectively engage one or more computing devices when the number of computing devices in the operating room exceeds a threshold number. In an example, a computing device (e.g., a wearable computing device and / or an ambient computing device) may indicate compatibility and / or adjust signal output to be compatible with an unknown computing system. A computing device may move through a range of viable frequencies and / or communication modalities to determine whether the computing device can potentially connect with a computing system it detects.

[0277] One or more computing systems and / or other computing devices can communicate with each other. In an example, a computing system may communicate with one or more computing devices (e.g., wearable computing devices). In an example, one or more digital devices may be present. A computing system may detect a surgeon through physical movement and / or automatic configuration. In an example, a computing system may set one or more instrument operating parameters based on, for example, detection of a technique used by the surgeon.

[0278] In an example, a computing system may detect a user, such as a surgeon, in an operating room (OR) based on physical movements by the user. For example, the surgeon may be wearing one or more computing devices, such as wearables, that may communicate with the computing system. The computing system may determine which action the user is performing based on information from the one or more computing devices. For example, the surgeon may wear a computing device (e.g., a wearable) on his / her wrist. The computing device may detect the surgeon holding an instrument, such as a surgical staple gun. The computing system may receive information from the computing device that the surgeon is holding the surgical staple gun. The computing system may determine one or more steps the surgeon may take.

[0279] The computing system may combine one or more static imaging techniques with continuous data monitoring (eg, from one or more measurement data).

[0280] Telemedicine can be interconnected with the computing system. In an example, telemedicine appointment scheduling can be based on intraoperative events. For example, based on intraoperative measurement data (e.g., parameters), one or more associated telemedicine providers can be queued and / or booked in the computing system for periodic follow-up.

[0281] A single or combination of intraoperative computing devices (e.g., sensors) can flag a patient if measurement data (e.g., monitored measurement data and / or variables) deviate from desired values. If a computing device flags based on measurement data, an HCP, such as a surgeon, may be alerted, for example, after a telemedicine follow-up is required in a given specialty. In an example, the telemedicine may be alerted (e.g., automatically) to set up the relevant follow-up.

[0282] In an example, a computing system and / or computing device may monitor measurement data, such as serum albumin. If the computing device detects a drop in serum albumin (e.g., below a preconfigured threshold serum albumin level), the computing device may prompt the need for scheduled post-operative nutritionist intervention. An update may be contingent on one or more appropriate criteria and / or set of criteria. In an example, an update may be contingent on one or more hardware capabilities of the computing system, such as processing power, bandwidth, resolution, etc. In an example, an update may be contingent on one or more software aspects, such as the purchase of specific software code. In an example, an update may be contingent on a purchased service tier. A service tier may represent a feature and / or set of features that a user is entitled to use in connection with a computer-implemented interactive surgical system. A service tier may be determined by a license code, an e-commerce server authentication interaction, a hardware key, a username / password combination, a biometric authentication interaction, a public / private key exchange interaction, etc.

[0283] The following is a non-exhaustive list of embodiments that are described above and / or shown in the drawings, and that may or may not be claimed below.

[0284] Example 1: An audio augmented reality (AR) computing system, comprising: a processor, receiving audio data from a sensing system in the operating room, the audio data including measurement data; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; An audio augmented reality (AR) computing system configured to:

[0285] The audio data, including the measurement data, may be associated with a user, such as a medical professional, an HCP (eg, a surgeon), or a patient.

[0286] Advantageously, audible information associated with particular measurement data relevant and / or important to the current surgical procedure may be adjusted or amplified (e.g., increased in volume), which may accordingly improve perception of the surgical procedure, the atmosphere in the OR, and / or interaction between HCPs.

[0287] Example 2: The audio AR computing system described in Example 1, wherein the adjustment instructions are received from the surgical computing system and include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating the insertion of a calming voice, a voice conversion instruction indicating the voice conversion of the voice data, or a sound source location instruction indicating the sound source location of the voice data.

[0288] Example 3: The adjustment instructions include the importance of the surgical steps, and the processor: Identify audio AR settings associated with the importance of surgical steps, Adjust the AR content according to your audio AR settings, The audio AR computing system according to Example 1 or Example 2, further configured as follows:

[0289] Advantageously, the AR content may be tailored according to the importance of the surgical steps which may improve awareness of the surgical procedure.

[0290] Example 4: The adjustment instructions optionally include audio information for critical surgical steps, and to adjust the generated AR content, the processor: Mute audio data from the sensing system; Optionally amplifying audio associated with audio information for critical surgical steps; The audio AR computing system according to any one of Examples 1 to 3, configured as follows:

[0291] Advantageously, the audibility of audio information that may be related to critical surgical steps may be increased, improving awareness of the surgical procedure.

[0292] Example 5: The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the processor: sending a frequency increase request to the sensing system based on the audio information including a frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease the frequency of transmitting audio data from the sensing system; 5. The audio AR computing system of example 4, configured to:

[0293] Advantageously, the audibility of audio information that may be related to critical surgical steps may be increased, improving awareness of the surgical procedure.

[0294] Example 6: The adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: identifying the relevance of the audio data from the sensing system to the surgical task indicated in the surgical task instructions; determining whether to block audio data from the sensing system based on the identified relevance to the surgical task indicated in the surgical task instructions, wherein the audio data from the sensing system is blocked if the audio data from the sensing system is unrelated to the surgical task indicated in the surgical task instructions, and / or the audio data from the sensing system is allowed if the audio data from the sensing system is related to the surgical task indicated in the surgical task instructions; 6. The audio AR computing system of any one of Examples 1 to 5, configured to:

[0295] Advantageously, the HCP may, for example, only follow audio data that is relevant to the surgical task being performed or to be performed, and thus, perception of the surgical procedure may be improved.

[0296] Example 7: A processor receiving an ambient noise level indication indicative of an ambient noise level in the operating room; sending a critical task instruction to the surgical computing system on the condition that the received ambient noise level is below a threshold ambient noise level, the critical task instruction indicating that a critical surgical task is to be performed; 7. The audio AR computing system of any one of Examples 1 to 6, configured to:

[0297] Advantageously, the system can determine that the HCP is about to perform a critical task and can send an alert to other HCPs that the next task involves a critical surgical task.

[0298] Example 8: Before adjusting the generated AR content, a processor: sending a user input request to the surgical computing system, the user input request requesting user input; receiving a user input associated with the user input request, wherein the AR content is further adjusted based on the user input; 8. The audio AR computing system of any one of Examples 1 to 7, configured to:

[0299] Advantageously, the user can adjust or tailor the AR content.

[0300] Example 9: The adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: Identifying audio AR settings associated with a surgical task; Adjusting the AR content according to the identified audio AR settings; 9. The audio AR computing system according to any one of Examples 1 to 8, configured as follows:

[0301] Advantageously, the AR content may be tailored according to the surgical task being performed or to be performed, thus improving the perception of the surgical procedure.

[0302] Example 10: A method for adjusting a surgical procedure, comprising: receiving a surgical instruction signal from a surgical device; receiving a surgical instruction signal from the surgical device; and receiving a surgical instruction signal from the surgical device; receiving second audio data from a second sensing system in the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on a user preference setting; Adjusting the AR content by increasing the volume of the selected preferred audio data 10. The audio AR computing system according to any one of Examples 1 to 9, configured as follows:

[0303] Advantageously, the system can handle multiple audio data sources according to user preferences.

[0304] Example 11: A method for adjusting a surgical procedure, comprising: receiving a surgical instruction signal from a surgical device; receiving a surgical instruction signal from the surgical device; and receiving a surgical instruction signal from the surgical device; receiving second audio data from a second sensing system in the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on a user preference setting; adjusting the AR content by reducing a volume of the first audio data from the first sensing system, provided that the second audio data from the second sensing system is selected as the preferred audio data; 10. The audio AR computing system according to any one of Examples 1 to 9, configured as follows:

[0305] Advantageously, the system can handle multiple audio data sources according to user preferences.

[0306] Example 12: A method for adjusting a surgical procedure, comprising: receiving a surgical instruction signal from a surgical device; and receiving a surgical instruction signal from the surgical device; and receiving a surgical instruction signal from the surgical device; the method comprising: receiving a surgical instruction signal from the surgical device; receiving second audio data from a second sensing system in the operating room; provided that the first audio data is preferred based on user preference settings; sending a first frequency increase request to the first sensing system to increase the frequency of transmitting the first audio data from the first sensing system; or transmitting a first frequency decrease request to the second sensing system to decrease the frequency of transmitting the second audio data from the second sensing system; Execute at least one of the following: provided that the second audio data is preferred based on user preference settings; sending a second frequency increase request to the second sensing system to increase the frequency of transmitting second audio data from the second sensing system; or transmitting a second frequency decrease request to the first sensing system to decrease the frequency of transmitting the first audio data from the first sensing system; perform at least one of the following: 10. The audio AR computing system according to any one of Examples 1 to 9, configured as follows:

[0307] Advantageously, the system can handle multiple audio data sources according to user preferences.

[0308] Example 13: The adjustment instructions include surgical step instructions indicating surgical steps, and the processor: receiving first audio data associated with a first healthcare professional (HCP) role in the operating room; receiving second audio data associated with a second HCP role in the operating room; determining whether first audio data associated with the first HCP is related to the surgical step indicated in the surgical step instructions and whether second audio data associated with the first HCP is related to the surgical step indicated in the surgical step instructions; adjusting the AR content by passing the first audio data associated with the first HCP and blocking the second audio data from the second HCP when it is determined that the first audio data associated with the first HCP is related to the surgical step and the second audio data associated with the first HCP is not related to the surgical step indicated in the surgical step instructions; 13. The audio AR computing system according to any one of Examples 1 to 12, configured as follows:

[0309] Advantageously, the system can handle multiple audio data sources according to the surgical steps.

[0310] Example 14: A computer-implemented method comprising: receiving audio data from a sensing system in the operating room, the audio data including measurement data; generating augmented reality (AR) content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; 11. A computer-implemented method comprising:

[0311] The audio data, including the measurement data, may be associated with a user, such as a medical professional, an HCP (eg, a surgeon), or a patient.

[0312] Advantageously, audible information associated with particular measurement data relevant and / or important to the current surgical procedure may be adjusted or amplified (e.g., increased in volume), which may accordingly improve perception of the surgical procedure, the atmosphere in the OR, and / or interaction between HCPs.

[0313] Example 15: The method described in Example 14, wherein the adjustment instructions are received from the surgical computing system and include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating the insertion of a calming voice, a voice conversion instruction indicating the voice conversion of the voice data, or a sound source location instruction indicating the sound source location of the voice data.

[0314] Example 16: The adjustment instructions include the importance of the surgical steps, and the method comprises: Identifying audio AR settings associated with the importance of a surgical step; and Adjusting AR content according to audio AR settings; The method of Example 14 or Example 15, comprising:

[0315] Advantageously, the AR content may be tailored according to the importance of the surgical steps which may improve awareness of the surgical procedure.

[0316] Example 17: To adjust the generated AR content, wherein the adjustment instructions optionally include audio information for critical surgical steps, a method comprises: mute audio data from the sensing system; Optionally amplifying audio associated with audio information for critical surgical steps; 17. The method of any one of Examples 14 to 16, comprising:

[0317] Advantageously, the audibility of audio information that may be related to critical surgical steps may be increased, improving awareness of the surgical procedure.

[0318] Example 18: The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the method further comprises: sending a frequency increase request to the sensing system based on the audio information including a frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease the frequency of transmitting audio data from the sensing system; The method of Example 17, comprising:

[0319] Advantageously, the audibility of audio information that may be related to critical surgical steps may be increased, improving awareness of the surgical procedure.

[0320] Example 19: A method comprising: receiving ambient noise in an operating room; Blocking received ambient noise; 19. The method of any one of Examples 14 to 18, comprising:

[0321] Advantageously, audio data may be received that may include ambient noise in the OR (eg, HCPs talking to each other, sounds of surgical instruments, etc.) that may be canceled and / or blocked.

[0322] Example 20: The adjustment instructions include surgical step instructions indicating surgical steps, and the method comprises: receiving first audio data associated with a first healthcare professional (HCP) role in an operating room; receiving second audio data associated with a second HCP role in the operating room; adjusting the AR content by allowing first audio data associated with a first HCP and blocking second audio data from a second HCP based on the surgical step instructions; 20. The method of any one of Examples 14 to 19, comprising:

[0323] Advantageously, the system can handle multiple audio data sources according to the surgical steps.

[0324] Example 21: A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of Examples 14 to 20.

[0325] The following is a non-exhaustive list of aspects described above and / or shown in the drawings, which may or may not be claimed below.

[0326] Aspect 1. An audio augmented reality (AR) computing system, comprising: a processor, receiving audio data from a sensing system in the operating room, the audio data including measurement data associated with a user; generating AR content based on the received audio data; obtaining an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the received adjustment instruction; An audio augmented reality (AR) computing system configured to:

[0327] Aspect 2. The audio AR computing system of Aspect 1, wherein the adjustment instructions are received from the surgical computing system and include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating the insertion of a calming voice, a voice conversion instruction indicating the voice conversion of the voice data, or a voice source location instruction indicating the sound source location of the voice data.

[0328] Aspect 3. The adjustment instructions include a surgical step importance, and the processor: Identify audio AR settings associated with the importance of surgical steps, Adjust the AR content according to your audio AR settings, 2. The audio AR computing system of claim 1, further configured as follows:

[0329] Aspect 4. The adjustment instructions include audio information for a critical surgical step, and to adjust the generated AR content, the processor: Mute audio data from the sensing system; amplifying audio associated with audio information for critical surgical steps; 2. The audio AR computing system of claim 1, configured as follows:

[0330] Aspect 5. The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the processor: sending a frequency increase request to the sensing system based on the audio information including a frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease the frequency of transmitting audio data from the sensing system; 5. The audio AR computing system of embodiment 4, configured to:

[0331] Aspect 6. The adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: identifying the relevance of the audio data from the sensing system to the surgical task indicated in the surgical task instructions; determining whether to block audio data from the sensing system based on the identified relevance to the surgical task indicated in the surgical task instructions, wherein the audio data from the sensing system is blocked if the audio data from the sensing system is associated with the surgical task indicated in the surgical task instructions; 2. The audio AR computing system of embodiment 1, configured to:

[0332] Aspect 7. A processor, receiving an ambient noise level indication indicative of an ambient noise level in the operating room; sending a critical task instruction to the surgical computing system on the condition that the received ambient noise level is below a threshold ambient noise level, the critical task instruction indicating that a critical surgical task is to be performed; 2. The audio AR computing system of embodiment 1, configured to:

[0333] Aspect 8. Before adjusting the generated AR content, the processor: sending a user input request to the surgical computing system, the user input request requesting user input; receiving a user input associated with the user input request, wherein the AR content is further adjusted based on the user input; 2. The audio AR computing system of embodiment 1, configured to:

[0334] Aspect 9. The adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: Identifying audio AR settings associated with a surgical task; Adjusting the AR content according to the identified audio AR settings; 2. The audio AR computing system of claim 1, configured as follows:

[0335] Aspect 10. A method for adjusting a surgical procedure, comprising: receiving a surgical instruction signal from a surgical device; receiving a surgical instruction signal from the surgical device; and receiving a surgical instruction signal from the surgical device; receiving second audio data from a second sensing system in the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on a user preference setting; Adjusting the AR content by increasing the volume of the selected preferred audio data 2. The audio AR computing system of claim 1, configured as follows:

[0336] Aspect 11. A method for adjusting a surgical procedure, comprising: receiving a surgical instruction signal from a surgical device; receiving a surgical instruction signal from the surgical device; and receiving a surgical instruction signal from the surgical device; receiving second audio data from a second sensing system in the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on a user preference setting; adjusting the AR content by reducing a volume of the first audio data from the first sensing system, provided that the second audio data from the second sensing system is selected as the preferred audio data; 2. The audio AR computing system of claim 1, configured as follows:

[0337] Aspect 12. A method for adjusting a surgical procedure, comprising: adjusting a surgical procedure; and adjusting a surgical procedure using a processor; wherein the audio data includes first audio data; the sensing system includes a first sensing system; and the adjustment instructions include user preference settings associated with a surgical procedure; and receiving second audio data from a second sensing system in the operating room; provided that the first audio data is preferred based on user preference settings; sending a first frequency increase request to the first sensing system to increase the frequency of transmitting the first audio data from the first sensing system; or transmitting a first frequency decrease request to the second sensing system to decrease the frequency of transmitting the second audio data from the second sensing system; Execute at least one of the following: provided that the second audio data is preferred based on user preference settings; sending a second frequency increase request to the second sensing system to increase the frequency of transmitting second audio data from the second sensing system; or transmitting a second frequency decrease request to the first sensing system to decrease the frequency of transmitting the first audio data from the first sensing system; perform at least one of the following: 2. The audio AR computing system of claim 1, configured as follows:

[0338] Aspect 13. The adjustment instructions include surgical step instructions indicating surgical steps, and the processor: receiving first audio data associated with a first healthcare professional (HCP) role in the operating room; receiving second audio data associated with a second HCP role in the operating room; determining whether first audio data associated with the first HCP is related to the surgical step indicated in the surgical step instructions and whether second audio data associated with the first HCP is related to the surgical step indicated in the surgical step instructions; adjusting the AR content by passing the first audio data associated with the first HCP and blocking the second audio data from the second HCP when it is determined that the first audio data associated with the first HCP is related to the surgical step and the second audio data associated with the first HCP is not related to the surgical step indicated in the surgical step instructions; 2. The audio AR computing system of claim 1, configured as follows:

[0339] Embodiment 14. A method comprising: receiving audio data from a sensing system in the operating room, the audio data including measurement data associated with a user; generating augmented reality (AR) content based on the received audio data; obtaining an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the received adjustment instruction; A method comprising:

[0340] Aspect 15. The method described in aspect 14, wherein the adjustment instructions are received from the surgical computing system and include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a sound source location instruction indicating the sound source location of the voice data.

[0341] Aspect 16. The adjustment instructions include the importance of the surgical steps, and the method further comprises: Identifying audio AR settings associated with the importance of a surgical step; and Adjusting AR content according to audio AR settings; 15. The method of embodiment 14, comprising:

[0342] Aspect 17. The adjustment instructions include audio information for critical surgical steps, and to adjust the generated AR content, the method comprises: mute audio data from the sensing system; amplifying audio associated with audio information for critical surgical steps; 15. The method of embodiment 14, comprising:

[0343] Aspect 18. The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the method further comprises: sending a frequency increase request to the sensing system based on the audio information including a frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease the frequency of transmitting audio data from the sensing system; 18. The method of embodiment 17, comprising:

[0344] Aspect 19. A method, comprising: receiving ambient noise in an operating room; Blocking received ambient noise; 15. The method of embodiment 14, comprising:

[0345] Aspect 20. The adjustment instructions include surgical step instructions indicating surgical steps, and the method comprises: receiving first audio data associated with a first healthcare professional (HCP) role in an operating room; receiving second audio data associated with a second HCP role in the operating room; adjusting the AR content by allowing first audio data associated with a first HCP and blocking second audio data from a second HCP based on the surgical step instructions; 15. The method of embodiment 14, comprising:

[0346] [Embodiment] (1) An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system in the operating room, the audio data including measurement data; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; An audio AR computing system configured to: (2) The audio AR computing system of embodiment 1, wherein the adjustment instructions include at least one of a surgical task instruction received from the surgical computing system and indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a sound source location instruction indicating the sound source location of the voice data. (3) the adjustment instructions include a surgical step importance, and the processor: identifying an audio AR setting associated with the importance of the surgical step; adjusting the AR content according to the audio AR settings; 3. The audio AR computing system of claim 1 or 2, further configured as follows: (4) the adjustment instructions optionally include audio information for critical surgical steps, and to adjust the generated AR content, the processor: mute the audio data from the sensing system; Optionally amplifying audio associated with said audio information for said critical surgical step; An audio AR computing system according to any one of embodiments 1 to 3, configured as follows: (5) The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the processor: sending a frequency increase request to the sensing system based on the audio information including the frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting the audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease a frequency of transmitting the audio data from the sensing system; An audio AR computing system as described in embodiment 4, configured to perform the following.

[0347] (6) the adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: identifying the relevance of the audio data from the sensing system to the surgical task indicated in the surgical task instructions; determining whether to block the audio data from the sensing system based on the identified relevance to the surgical task indicated in the surgical task instructions, wherein the audio data from the sensing system is blocked if the audio data from the sensing system is unrelated to the surgical task indicated in the surgical task instructions, and / or the audio data from the sensing system is allowed if the audio data from the sensing system is related to the surgical task indicated in the surgical task instructions; An audio AR computing system described in any one of embodiments 1 to 5, configured to perform the following. (7) The processor: receiving an ambient noise level indication indicative of an ambient noise level in the operating room; sending a critical task instruction to a surgical computing system, on the condition that the received ambient noise level is below a threshold ambient noise level, the critical task instruction indicating that a critical surgical task is to be performed; An audio AR computing system described in any one of embodiments 1 to 6, configured to perform the following. (8) before adjusting the generated AR content, the processor: sending a user input request to a surgical computing system, the user input request requesting user input; receiving the user input associated with the user input request, wherein the AR content is further adjusted based on the user input; and An audio AR computing system as described in any one of embodiments 1 to 7, configured to perform the following. (9) The adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: Identifying an audio AR setting associated with the surgical task; adjusting the AR content according to the identified audio AR settings; An audio AR computing system according to any one of embodiments 1 to 8, configured as follows: (10) The audio data includes first audio data, the sensing system includes a first sensing system, and the adjustment instructions include user preference settings associated with a surgical procedure, and the processor: receiving second audio data from a second sensing system in the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on the user preference setting; Adjusting the AR content by increasing the volume of the selected preferred audio data. An audio AR computing system as described in any one of embodiments 1 to 9, configured as follows.

[0348] (11) The audio data includes first audio data, the sensing system includes a first sensing system, and the adjustment instructions include user preference settings associated with a surgical procedure, and the processor: receiving second audio data from a second sensing system in the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on the user preference setting; adjusting the AR content by reducing a volume of the first audio data from the first sensing system, provided that the second audio data from the second sensing system is selected as the preferred audio data; An audio AR computing system as described in any one of embodiments 1 to 9, configured as follows. (12) The audio data includes first audio data, the sensing system includes a first sensing system, and the adjustment instructions include user preference settings associated with a surgical procedure, and the processor: receiving second audio data from a second sensing system in the operating room; provided that the first audio data is preferred based on the user preference setting; sending a first frequency increase request to the first sensing system to increase the frequency of transmitting the first audio data from the first sensing system; or sending a first frequency decrease request to the second sensing system to decrease the frequency of transmitting the second audio data from the second sensing system; Execute at least one of the following: provided that the second audio data is preferred based on the user preference setting; sending a second frequency increase request to the second sensing system to increase the frequency of transmitting the second audio data from the second sensing system; or sending a second frequency decrease request to the first sensing system to decrease the frequency of transmitting the first audio data from the first sensing system; perform at least one of the following: An audio AR computing system as described in any one of embodiments 1 to 9, configured as follows. (13) The adjustment instructions include surgical step instructions indicating surgical steps, and the processor: receiving first audio data associated with a first healthcare professional (HCP) role in the operating room; receiving second audio data associated with a second HCP role in the operating room; determining whether the first voice data associated with the first HCP is related to the surgical step indicated in the surgical step instructions and whether the second voice data associated with the first HCP is related to the surgical step indicated in the surgical step instructions; adjusting the AR content by passing the first audio data associated with the first HCP and blocking the second audio data from the second HCP when it is determined that the first audio data associated with the first HCP is related to the surgical step and the second audio data associated with the first HCP is not related to the surgical step indicated in the surgical step instructions; An audio AR computing system as described in any one of embodiments 1 to 12, configured as follows. (14) A computer-implemented method comprising: receiving audio data from a sensing system in the operating room, the audio data including measurement data; generating augmented reality (AR) content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and 11. A computer-implemented method comprising: (15) The method of embodiment 14, wherein the adjustment instructions include at least one of a surgical task instruction received from a surgical computing system and indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a sound source location instruction indicating a sound source location of the voice data.

[0349] (16) The adjustment instructions include the importance of a surgical step, and the method further comprises: identifying an audio AR setting associated with the importance of the surgical step; adjusting the AR content according to the audio AR settings; and 16. The method of embodiment 14 or 15, comprising: (17) The adjustment instructions optionally include audio information for critical surgical steps, and to adjust the generated AR content, the method further comprises: muting the audio data from the sensing system; and Optionally amplifying audio associated with said audio information for said critical surgical step; 17. The method of any of embodiments 14 to 16, comprising: (18) The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the method further comprises: sending a frequency increase request to the sensing system based on the audio information including the frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting the audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease a frequency of transmitting the audio data from the sensing system; 18. The method of embodiment 17, comprising: (19) The method further comprises: receiving ambient noise in the operating room; blocking the received ambient noise; 19. The method of any of embodiments 14 to 18, comprising: (20) The adjustment instruction includes a surgical step instruction indicating a surgical step, and the method further comprises: receiving first audio data associated with a first healthcare professional (HCP) role in the operating room; receiving second audio data associated with a second HCP role in the operating room; adjusting the AR content by allowing the first audio data associated with the first HCP and blocking the second audio data from the second HCP based on the surgical step instructions; 20. The method of any of embodiments 14 to 19, comprising:

[0350] (21) A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of embodiments 14 to 20.

Claims

1. 1. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and The adjustment instructions include a surgical step importance, and the processor: identifying an audio AR setting associated with the importance of the surgical step; adjusting the AR content according to the audio AR settings; The audio AR computing system is further configured as follows.

2. 2. The audio AR computing system of claim 1, wherein the adjustment instructions include at least one of a surgical task instruction received from a surgical computing system and indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a sound source location instruction indicating a sound source location of the voice data.

3. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the adjustment instructions include audio information for a critical surgical step, and to adjust the generated AR content, the processor: mute the audio data from the sensing system; amplifying the audio associated with the audio information for the critical surgical step; An audio AR computing system configured as follows.

4. The audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the processor: sending a frequency increase request to the sensing system based on the audio information including the frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting the audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease a frequency of transmitting the audio data from the sensing system; The audio AR computing system of claim 3 configured to:

5. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: identifying the relevance of the audio data from the sensing system to the surgical task indicated in the surgical task instructions; determining whether to block the audio data from the sensing system based on the identified relevance to the surgical task indicated in the surgical task instructions, wherein the audio data from the sensing system is blocked on the condition that the audio data from the sensing system is unrelated to the surgical task indicated in the surgical task instructions, and / or the audio data from the sensing system is allowed on the condition that the audio data from the sensing system is related to the surgical task indicated in the surgical task instructions; An audio AR computing system configured to:

6. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the processor: receiving an ambient noise level indication indicative of an ambient noise level in the operating room; sending a critical task instruction to a surgical computing system, on the condition that the received ambient noise level is below a threshold ambient noise level, the critical task instruction indicating that a critical surgical task is to be performed; An audio AR computing system configured to:

7. Before adjusting the generated AR content, the processor: sending a user input request to a surgical computing system, the user input request requesting user input; receiving the user input associated with the user input request, wherein the AR content is further adjusted based on the user input; 7. The audio AR computing system of claim 1, configured to:

8. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the adjustment instructions include surgical task instructions indicating a surgical task being performed or to be performed, and the processor: Identifying an audio AR setting associated with the surgical task; adjusting the AR content according to the identified audio AR settings; An audio AR computing system configured as follows.

9. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the audio data includes first audio data, the sensing system includes a first sensing system, the adjustment instructions include user preference settings associated with a surgical procedure, and the processor: receiving second audio data from a second sensing system configured to measure sounds within the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on the user preference setting; Adjusting the AR content by increasing the volume of the selected preferred audio data. It is structured as follows:

10. An audio AR computing system, wherein the second audio data includes one or more of a sound associated with a user, ambient noise in an operating room, audible feedback, and audible information measured by the second sensing system.

10. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the audio data includes first audio data, the sensing system includes a first sensing system, the adjustment instructions include user preference settings associated with a surgical procedure, and the processor: receiving second audio data from a second sensing system configured to measure sounds within the operating room; selecting preferred audio data between the first audio data from the first sensing system and the second audio data from the second sensing system based on the user preference setting; adjusting the AR content by reducing a volume of the first audio data from the first sensing system, provided that the second audio data from the second sensing system is selected as the preferred audio data; It is structured as follows:

10. An audio AR computing system, wherein the second audio data includes one or more of a sound associated with a user, ambient noise in an operating room, audible feedback, and audible information measured by the second sensing system.

11. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and the audio data includes first audio data, the sensing system includes a first sensing system, the adjustment instructions include user preference settings associated with a surgical procedure, and the processor: receiving second audio data from a second sensing system configured to measure sounds within the operating room; provided that the first audio data is preferred based on the user preference settings; sending a first frequency increase request to the first sensing system to increase the frequency of transmitting the first audio data from the first sensing system; or transmitting a first frequency decrease request to the second sensing system to decrease the frequency of transmitting the second audio data from the second sensing system; and performing at least one of provided that the second audio data is preferred based on the user preference settings; sending a second frequency increase request to the second sensing system to increase the frequency of transmitting the second audio data from the second sensing system; or sending a second frequency decrease request to the first sensing system to decrease the frequency of transmitting the first audio data from the first sensing system; Execute at least one of: It is structured as follows:

10. An audio AR computing system, wherein the second audio data includes one or more of a sound associated with a user, ambient noise in an operating room, audible feedback, and audible information measured by the second sensing system.

12. An audio augmented reality (AR) computing system, comprising: a processor, the processor comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; and The adjustment instructions include surgical step instructions indicating surgical steps, and the processor: receiving first audio data associated with a first HCP role in the operating room; receiving second audio data associated with a second HCP role in the operating room; determining whether the first voice data associated with the first HCP is related to the surgical step indicated in the surgical step instructions, and whether the second voice data associated with the second HCP is related to the surgical step indicated in the surgical step instructions; adjusting the AR content by passing the first audio data associated with the first HCP and blocking the second audio data from the second HCP when it is determined that the first audio data associated with the first HCP is related to the surgical step and the second audio data associated with the second HCP is not related to the surgical step indicated in the surgical step instructions; An audio AR computing system configured as follows.

13. 1. A computer-implemented method comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; Including, the adjustment instructions include a surgical step importance, and the computer-implemented method further comprises: identifying an audio AR setting associated with the importance of the surgical step; adjusting the AR content according to the audio AR settings; 11. A computer-implemented method comprising:

14. 14. The computer-implemented method of claim 13, wherein the adjustment instructions are received from a surgical computing system and include at least one of a surgical task instruction indicating a surgical task being performed or to be performed, a task importance instruction indicating the importance of the surgical task, a voice insertion instruction indicating insertion of a calming voice, a voice conversion instruction indicating voice conversion of the voice data, or a sound source location instruction indicating a sound source location of the voice data.

15. A computer-implemented method comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; Including, The adjustment instructions include audio information for critical surgical steps, and to adjust the generated AR content, the computer-implemented method further comprises: muting the audio data from the sensing system; and amplifying the audio associated with the audio information for the critical surgical step; 11. A computer-implemented method comprising:

16. and wherein the audio information for the critical surgical step further includes at least one of an increase frequency instruction or a decrease frequency instruction, and the computer-implemented method further comprises: sending a frequency increase request to the sensing system based on the audio information including the frequency increase instruction, the frequency increase request including a request to increase the frequency of transmitting the audio data from the sensing system; or transmitting a frequency decrease request to the sensing system based on the audio information including the frequency decrease instruction, the frequency decrease request including a request to decrease a frequency of transmitting the audio data from the sensing system; 16. The computer-implemented method of claim 15, comprising:

17. A computer-implemented method comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; Including, The computer-implemented method comprises: receiving ambient noise in the operating room; blocking the received ambient noise; 11. A computer-implemented method comprising:

18. A computer-implemented method comprising: receiving audio data from a sensing system configured to measure sounds within an operating room, the audio data including one or more of sounds associated with a user, ambient noise in the operating room, audible feedback, and audible information measured by the sensing system; generating AR content based on the received audio data; obtaining or receiving an adjustment instruction indicating adjustment information for the AR content; adjusting the generated AR content based on the adjustment instruction; Including, the adjustment instructions include surgical step instructions indicating surgical steps, and the computer-implemented method further comprises: receiving first audio data associated with a first HCP role in the operating room; receiving second audio data associated with a second HCP role in the operating room; adjusting the AR content by allowing the first audio data associated with the first HCP and blocking the second audio data from the second HCP based on the surgical step instructions; 11. A computer-implemented method comprising:

19. A computer readable medium comprising instructions that, when executed by a computer, cause the computer to perform the computer-implemented method of any one of claims 13 to 18.

Images