Verification of controlled access for medical professionals

Abstract

The computing system may identify a surgical instrument for a surgical procedure in an operating room (OR). The computing system may detect a control input by a healthcare professional (HCP) to control the surgical instrument. The computing system may determine an access control level of the HCP associated with the surgical instrument. The computing system may determine whether the HCP has authorization to control the surgical instrument. If the computing system determines that the HCP is not authorized to control the surgical instrument based on the access control level associated with the HCP, the computing system may block the control input by the HCP. If the computing system determines that the HCP is authorized to control the surgical instrument based on the access control level associated with the HCP, the computing system may enable the control input by the HCP to control the surgical instrument.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 194,675, filed May 28, 2021, the disclosure of which is hereby incorporated by reference in its entirety.

[0002] This application is related to the following applications filed concurrently, the content of each of which is hereby incorporated by reference in its entirety. · U.S. Patent Application No. 17 / 335,738, titled METHOD OF MONITORING AND ANALYZING SURGICAL PROCEDURES, Attorney Docket No. END9339USNP1.

Background Art

[0003] The success of a surgery depends on the expertise of several types of operating room (OR) team members. The roles of the operating room staff are diverse. Further, the surgeon makes critical decisions involved in directing the course of the procedure. The surgeon may perform the incisions involved in the surgery. An anesthesiologist or nurse anesthetist may be responsible for safely administering anesthesia to the patient before surgery, monitoring the patient during surgery, and confirming that the patient has safely awakened from anesthesia after surgery. A circulating technician may bring the patient into the operating room, tie the surgical gowns of the surgeon and other personnel, deliver additional supplies such as instruments and medications that are needed, and document the surgery. Instrument processing specialists may sterilize the instruments before and after surgery, keep the surgical field organized during surgery, and provide the surgeon with the necessary instruments. A registered nurse may perform duties typically associated with surgical technologists, including serving as a scrub nurse and an instrument processing nurse. In addition, the nurse may serve as the surgeon's first assistant. The traffic of people in the OR has been associated as a cause of surgical site infections.

Summary of the Invention

Means for Solving the Problems

[0004] The computer implementation method may include identifying a surgical instrument associated with a surgical procedure in an operating room (OR), detecting a control input from a healthcare professional (HCP) to control the surgical instrument, determining the HCP's access control level associated with the surgical instrument, and determining whether to enable the control input based on the HCP's access control level associated with the surgical instrument, thereby blocking the HCP's control input to control the surgical instrument based on the determination that the HCP's access control level associated with the surgical instrument does not authorize the HCP to enable the control input to control the surgical instrument, and enabling the HCP's control input to control the surgical instrument based on the determination that the HCP's access control level associated with the surgical instrument authorizes the HCP to enable the control input to control the surgical instrument.

[0005] The computer implementation method can provide the technical effect of preventing accidental activation of surgical instruments by authorizing or disauthorizing the HCP to enable control inputs for controlling surgical instruments. Such accidental activation of surgical instruments may occur, for example, while the HCP is cleaning and / or disposing of the instruments.

[0006] An access control level associated with a surgical instrument may include a first access control level for controlling the surgical instrument and a second access control level for controlling the surgical instrument, wherein the first access control level may include at least one of turning the surgical instrument on or turning the surgical instrument off, and the second access control level may include at least one of increasing the energy level associated with the surgical instrument or decreasing the energy level associated with the surgical instrument.

[0007] Computerized implementation methods can provide the technical benefit of allowing hierarchical access to surgical instruments by permitting some, though not all, of the controls for a specified HCP.

[0008] The control input may be a first control input, and the HCP may be a first HCP, and the method may include detecting a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to turn the surgical instrument on or off; determining the access control level of the second HCP associated with the surgical instrument; and enabling the second control input to turn the surgical instrument on or off based on the determination that the access control level of the second HCP associated with the surgical instrument authorizes the second HCP to turn the surgical instrument on or off.

[0009] The control input may be a first control input, and the HCP may be a first HCP; the method may include detecting a second control input by a second HCP for controlling a surgical instrument, the second control input being configured to change the energy level associated with the surgical instrument; determining the access control level of the second HCP associated with the surgical instrument; and blocking the control input for controlling the surgical instrument based on the determination that the access control level of the second HCP associated with the surgical instrument does not authorize the second HCP to change the energy level associated with the surgical instrument.

[0010] The computer implementation method may include sending an alert to at least one of the devices associated with the HCP or a display in the OR, the alert may include at least one of the following: a notification that control input by the HCP has been blocked, or access control level information assigned to the HCP.

[0011] Computerized implementation methods can provide the technical benefit of keeping HCPs better informed about surgical instruments within the OR, and can allow HCPs to modify the access control levels of other HCPs.

[0012] The computer implementation method may include: monitoring the movement associated with a first HCP; determining that the first HCP is in proximity to the operating table; sending an access control level adjustment inquiry message to a second HCP based on the determination that the first HCP is in proximity to the operating table, wherein the access control level adjustment inquiry message is configured to prompt the HCP to indicate whether the access control level of the HCP associated with surgical instruments needs adjustment based on the proximity of the first HCP to the operating table; receiving an access control level adjustment request from the second HCP in response to the access control level adjustment inquiry message; and adjusting the access control level associated with the first HCP based on the access control level adjustment request.

[0013] The computer implementation method can provide the technical benefit of allowing more efficient surgical procedures by enabling a second HCP to supervise and adjust the access control level of the first HCP.

[0014] The computer implementation method may include sending an access control level adjustment notification to the HCP, which notifies that the access control level of the HCP associated with the surgical instrument has been adjusted by a second HCP.

[0015] Computerized implementation methods can offer the technical benefit of ensuring that HCPs within an OR are better informed about the actions of other HCPs, which can lead to more efficient surgical procedures.

[0016] The method may include identifying the current surgical step in a surgical procedure, determining whether to adjust the access control level associated with the HCP based on the identified current surgical step, and adjusting the access control level of the HCP associated with the surgical instrument so that the HCP can control the surgical instrument during the current surgical step in the surgical procedure, based on the determination to adjust the access control level of the HCP associated with the surgical instrument.

[0017] The computer implementation method can provide the technical effect of preventing or allowing the HCP to make adjustments to surgical instruments according to the current surgical procedure, thereby improving patient safety and / or surgical outcomes.

[0018] The method may include identifying the energized state of a surgical instrument and determining whether to adjust the access control level of the HCP associated with the surgical instrument based on the identified energized state of the surgical instrument, wherein, based on the identified energized state of the surgical instrument being in a high energized state, the method may include adjusting the access control level of the HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument, and based on the identified energized state of the surgical instrument being in a low energized state, the method may include adjusting the access control level of the HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument.

[0019] The computer implementation method can provide a technical benefit, for example, by preventing inadvertent control of surgical instruments by blocking the control input from the scrub nurse when it is determined that the surgical instrument is in a highly energized state.

[0020] The method may include monitoring a biomarker associated with an HCP, the biomarker may include at least one of fatigue levels or stress levels, and determining whether to adjust the access control level of the HCP associated with a surgical instrument based on the monitored biomarker associated with the HCP, and based on the monitored biomarker associated with the HCP indicating that the HCP has an increased level of at least one of fatigue levels or stress levels, the method may include adjusting the access control level of the HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument, and based on the monitored biomarker associated with the HCP indicating that the HCP has an acceptable level of at least one of fatigue levels or stress levels, the method may include adjusting the access control level of the HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument.

[0021] Computerized implementation methods can provide technical benefits such as improving patient safety and / or surgical outcomes based on HCP biomarkers.

[0022] A computing system may include a processor configured to perform one of the methods described above.

[0023] The data processing device may be equipped with means for carrying out any of the methods described above.

[0024] The system may comprise the aforementioned computing system and surgical instruments that are communicatively connected to the computer system.

[0025] The system may include the aforementioned data processing device and surgical instruments that are communicatively connected to the data processing device.

[0026] A computer-readable medium can have instructions that, when executed by a computer, implement any of the above methods.

[0027] A computing system can identify surgical instruments associated with a surgical procedure in an operating room (OR). During a surgical operation, the computing system can detect control inputs by a healthcare professional (HCP) for controlling the surgical instrument. The control inputs by the HCP can include turning on / off the surgical instrument, increasing / decreasing the power-on state of the surgical instrument, etc. When the computing system detects a control input by the HCP, the computing system can determine whether the HCP is authorized to provide the control input to control the surgical instrument. For example, the computing system can determine the access control level of the HCP associated with the surgical instrument. This control level can provide information regarding whether the HCP is authorized to control the surgical instrument. In an example, the control level can indicate that the HCP is not authorized to control the surgical instrument. In an example, the control level can indicate that the HCP is authorized to control the surgical instrument. The access control level can provide information regarding whether the HCP has full control for controlling the surgical instrument. The access control level can provide information regarding whether the HCP has partial control (e.g., limited to turning on / off the surgical instrument) for controlling the surgical instrument. For example, the access control level of the HCP can be hierarchical. A lower-level access control level can permit / confirm that the HCP can turn on / off the surgical instrument. A higher-level access control can permit / confirm that the HCP can adjust the power-on state of the surgical instrument.

[0028] If the computing system determines that the HCP is authorized to control the surgical instrument, the computing system can enable the control input by the HCP. For example, if the computing system determines that the HCP is not authorized to control the surgical instrument, the computing system can block the control input by the HCP. If the computing system determines that the HCP is authorized to control the surgical instrument, the computing system can enable the control input by the HCP.

[0029] The computing system can send an alert to the HCP. The alert can notify the HCP whether the HCP is authorized or not authorized to control the surgical instrument. The alert can also notify the HCP whether the control input has been enabled or blocked, for example, based on the control access level associated with the HCP. The alert can be one or more of a virtual alert, an audible alert, a tactile alert, or an augmented reality alert, or can include them.

[0030] The computing system can adjust the control access level of the HCP. The computing system can adjust the control level of the HCP based on the proximity to and / or within the virtual boundary of the access restricted area. For example, if the HCP is not authorized to control the surgical instrument and the computing system determines that the HCP is within the virtual boundary area of the operating table and / or the virtual boundary area of the surgeon, the computing system can adjust the control access level of the HCP and provide partial authorization to control the surgical instrument (e.g., turn the surgical instrument on / off).

[0031] The computing system can adjust the control access levels of an HCP based on requests from other HCPs within the OR. For example, if the computing system blocks control input from an HCP such as a scrub nurse, the computing system can send a message to another HCP in the OR, such as a surgeon. The message may be or may contain an access control level adjustment message. If another HCP, such as a surgeon, determines that a scrub nurse should be able to control surgical instruments, for example, be able to turn on surgical instruments before handing them to the surgeon, the surgeon can send an access control level adjustment request to the computing system. Based on the receipt of the access control level adjustment request from the surgeon, the computing system can adjust the access control of the scrub nurse. The computing system can send notifications to an HCP, such as an access control level adjustment notification. The notification may inform the HCP that the access control level associated with the HCP has been adjusted by another HCP, such as a surgeon.

[0032] The computing system can adjust the control access level of the HCP based on the surgical process in a surgical procedure. The computing system can identify the current surgical process in a surgical procedure. Based on the current surgical process, the computing system can adjust the control access level of the HCP. For example, to prevent accidental control of surgical instruments, the computing system can adjust the control access level of the HCP when it determines that the current surgical process corresponds to the operation of a surgical instrument. The computing system can block control input from the HCP in other surgical processes.

[0033] The computing system can adjust the HCP control access level based on the energized state of the surgical instrument. The computing system can determine whether the surgical instrument is in a low-energy or high-energy state. If the surgical instrument is in a high-energy state, the computing system can block the HCP control input. If the surgical instrument is in a low-energy state, the computing system can enable the HCP control input.

[0034] The computing system can adjust the control access level of the HCP based on biomarker measurements associated with the HCP. For example, the computing system can monitor one or more biomarkers of the HCP. If the computing system detects an increase and / or rise in the HCP's stress or fatigue level, it can adjust the HCP's control access level by blocking the HCP's control input. If the computing system detects an acceptable stress and / or fatigue level for the HCP, it can adjust the HCP's control access level by enabling the HCP's control input. [Brief explanation of the drawing]

[0035] [Figure 1] This is a block diagram of a computer-implemented healthcare personnel (HCP) monitoring system. [Figure 2] An example of an HCP monitoring system in a surgical operating room is shown. [Figure 3] This shows an exemplary surgical hub paired with various systems. [Figure 4] This describes a surgical data network having a set of communication surgical hubs configured to connect to a set of sensing systems, an environmental sensing system, a set of devices, and so on. [Figure 5]This exhibits an exemplary computer-implemented bidirectional surgical system that may be part of an HCP monitoring system. [Figure 6] This shows a logic diagram of the control system for surgical instruments. [Figure 7] An exemplary sensing system having a sensor unit and a data processing and communication unit is shown. [Figure 8] This example shows an exemplary timeline of a surgical procedure that illustrates how to adjust the operating parameters of a surgical device based on the surgeon's biomarker level. [Figure 9] This is a block diagram of a computer-implemented bidirectional HCP monitoring system. [Figure 10] An exemplary surgical system is shown, which includes surgical instruments that communicate with a console or portable device. [Figure 11] This is a diagram illustrating an exemplary situational awareness surgical system. [Figure 12] Examples of one or more virtual boundaries associated with restricted access in an operating room (OR) are shown. [Figure 13] Examples of one or more virtual boundaries associated with restricted access within an OR are shown. [Figure 14] Examples of constant virtual boundaries, selective virtual boundaries, or adaptive virtual boundaries within an OR are shown. [Figure 15] This shows an exemplary virtual boundary associated with surgical instruments. [Figure 16] This provides an illustrative notification from a computing system regarding unauthorized energy level changes in surgical instruments. [Figure 17] This example flow shows how to monitor HCP behavior against the virtual boundary of an access-restricted area within an OR. [Figure 18] This shows an example flow for HCP control access verification within an Operations Research Unit (OR). [Modes for carrying out the invention]

[0036] Figure 1 is a block diagram of a computer-implemented HCP monitoring system 20000. An exemplary HCP monitoring system, such as HCP monitoring system 20000, may include one or more HCP monitoring systems (e.g., HCP monitoring subsystems) 20002, 20003, and 20004. For example, HCP monitoring system 20002 may include a computer-implemented bidirectional surgical system. For example, HCP monitoring system 20002 may include at least one surgical hub 20006 that communicates with a cloud computing system 20008, for example, as shown in Figure 2. The HCP monitoring system may include at least one surgical hub 20006 or a computing device 20016 that communicates with the cloud computing system 20008. The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. An exemplary HCP monitoring system 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent devices 20014, a human interface system 20012, etc. A human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include one or more devices used to measure one or more environmental attributes, for example, as further described in Figure 2. The robotic system 20013 may include multiple devices used to perform surgical procedures, for example, as further described in Figure 2.

[0037] The HCP monitoring system 20002 may communicate with a remote server 20009, which is part of the cloud computing system 20008. In one embodiment, the HCP monitoring system 20002 may communicate with the remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one embodiment, the patient sensing system may communicate directly with the remote server 20009. The HCP monitoring system 20002 and / or its components may communicate with the remote server 20009 via a cellular transmission / reception point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE-Advanced (LTE-A), New Radio (NR), or 5G.

[0038] The surgical hub 20006 may have collaborative interaction with one or more means of displaying information from images from a laparoscope, one or more other smart devices, and one or more sensing systems 20011. The surgical hub 20006 may interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 may be configured to collect measurement data from one or more sensing systems 20011 and to send notification or control messages to one or more sensing systems 20011. The surgical hub 20006 may transmit and / or receive information, including notification information, to and from a human interface system 20012. The human interface system 20012 may include one or more human interface devices (HIDs). The surgical hub 20006 can transmit and / or receive notification or control information to audio equipment, displays, and / or control information to various devices that communicate with the surgical hub.

[0039] For example, sensing system 20001 may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, as discussed in Figure 1. One or more sensing systems 20001 may measure data related to various biomarkers. One or more sensing systems 20001 may measure biomarkers using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. One or more sensors may measure biomarkers using one or more sensing techniques such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary, potentiometric measurement, and current measurement, as described herein.

[0040] Biomarkers measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, hydrogen blood potential, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous system tone, circadian rhythm, and / or menstrual cycle.

[0041] Biomarkers may relate to physiological systems, including but not limited to behavioral and psychological systems, cardiovascular systems, renal systems, cutaneous systems, nervous systems, gastrointestinal systems, respiratory systems, endocrine systems, immune systems, tumors, musculoskeletal systems, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by a computer-implemented patient and HCP monitoring system 20000. Information from biomarkers may be determined and / or used by the computer-implemented patient and HCP monitoring system 20000 to improve the system and / or improve patient outcomes, for example. One or more sensing systems 20001, biomarkers 20005, and physiological systems are described in more detail in U.S. Patent Application No. 17 / 156,287, filed on January 22, 2021, entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS" (Agent Reference Number END9290USNP1). That disclosure is incorporated herein by reference in its entirety.

[0042] Figure 2 shows an example of an HCP monitoring system 20002 in a surgical operating room. As shown in Figure 2, the patient is operated on by one or more medical professionals (HCPs). The HCPs are monitored by one or more HCP sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding them may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors that may be deployed in the operating room. The HCP sensing systems 20020 and the environmental sensing systems communicate with a surgical hub 20006 and may further communicate with one or more cloud servers 20009 of a cloud computing system 20008, as shown in Figure 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as the location of the HCPs in the operating room, the movement of the HCPs, ambient noise in the operating room, and temperature / humidity in the operating room.

[0043] As shown in Figure 2, the main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned in the sterile field so that they are visible to the operator on the operating table 20024. In addition, a visualization / notification tower 20026 is positioned outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human interactive device (HID) 20027 and a second non-sterile HID 20029, which face opposite each other. The HIDs may be displays or displays with touchscreens that allow humans to interface directly with the HIDs. A human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to coordinate the flow of information to operators inside and outside the sterile field. In one embodiment, the surgical hub 20006 allows the HID (e.g., main HID 20023) to display notifications and / or information relating to the patient and / or surgical procedure. In one embodiment, the surgical hub 20006 can prompt and / or receive input from a person in a sterile field or non-sterile area. In one embodiment, the surgical hub 20006 allows the HID to display snapshots of the surgical site recorded by the imaging device 20030 on a non-sterile HID 20027 or 20029 while maintaining live video of the surgical site on the main HID 20023. The snapshots on the non-sterile display 20027 or 20029 may, for example, allow a non-sterile operator to perform diagnostic procedures related to the surgical procedure.

[0044] In one embodiment, the surgical hub 20006 may be configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 20026 to a main display 20023 in the sterile field, which can then be viewed by a sterile operator at the operating table. In one embodiment, the input may take the form of modifications to snapshots displayed on non-sterile displays 20027 or 20029, which can then be sent to the main display 20023 by the surgical hub 20006.

[0045] Referring to Figure 2, surgical instrument 20031 is used as part of the HCP monitoring system 20002 in a surgical procedure. Hub 20006 may also be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, this is described in U.S. Patent Application Publication 2019 / 0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 20026 may be sent by Hub 20006 to a surgical instrument display in the sterile field, where it can be viewed by the operator of surgical instrument 20031. An exemplary surgical instrument suitable for use with the HCP monitoring system 20002 is, for example, U.S. Patent Application Publication No. 2019 / 0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," described under the heading "Surgical Instrument Hardware," the disclosure of which is incorporated herein by reference in its entirety.

[0046] Figure 2 shows an example of an HCP monitoring system 20002 used to perform a surgical procedure on a patient lying on an operating table 20024 in a surgical operating room 20035. A robotic system 20034 may be used as part of the HCP monitoring system 20002 in a surgical procedure. The robotic system 20034 may include a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. While the surgeon views the surgical site through the surgeon's console 20036, the patient-side cart 20032 can operate at least one detachably connected surgical tool 20037 through a minimally invasive incision in the patient's body. Images of the surgical site may be acquired by a medical imaging device 20030, which can be operated by the patient-side cart 20032 to change the orientation of the imaging device 20030. The robot hub 20033 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 20036.

[0047] Other types of robotic systems can be readily adapted for use with the HCP monitoring system 20002. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019 / 0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0048] Various examples of cloud-based analytical methods implemented by Cloud Computing System 20008 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019 / 0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0049] In various embodiments, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0050] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0051] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The human eye typically responds to wavelengths in air from about 380 nm to about 750 nm.

[0052] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0053] In various embodiments, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, gastroscopy (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0054] The imaging device may utilize multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging can extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019 / 0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is necessary in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible level of sterility for all medical devices and instruments. Part of the sterilization process includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including imaging devices and their accessories and components. It will be understood that the sterile field may be considered a specific area deemed free of microorganisms, such as within a tray or on a sterile towel, or it may be considered the area immediately surrounding a patient ready for surgical treatment. The sterile field may include cleaned team members in appropriate clothing, as well as all equipment and restraints within that area.

[0055] The wearable sensing system 20011 shown in Figure 1 may include one or more sensing systems, for example, an HCP sensing system 20020 as shown in Figure 2. The HCP sensing system 20020 may include a sensing system for monitoring and detecting a set of physical and / or physiological conditions of a healthcare professional (HCP). An HCP is generally one or more healthcare professionals assisting a surgeon or other healthcare service provider. In one embodiment, the sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In one embodiment, the sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the magnitude and frequency of the tremors. The sensing system 20020 may transmit the measured data associated with the set of biomarkers and the data associated with the surgeon's physical condition to a surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the position of the HCP's hand / body. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the surgical field. Other environmental sensing devices may include, for example, a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the surgical field. The surgical hub 20006 may, independently or in communication with a cloud computing system, use the surgeon biomarker measurement data and / or environmental sensing information to, for example, correct the control algorithm for handheld instruments or the average delay of a robotic interface to minimize tremor. In one embodiment, the HCP sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit the measurement data associated with the surgeon biomarkers to the surgical hub 20006.The HCP sensing system 20020 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-power wireless Personal Area Network (6LoWPAN), and Wi-Fi. Surgical biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels related to the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention levels, etc.

[0056] The surgical hub 20006 can adaptively control one or more surgical instruments 20031 using surgical biomarker measurement data associated with the HCP. For example, the surgical hub 20006 may transmit a control program to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may transmit a control program based on contextual awareness and / or circumstances relating to the importance or criticality of the task. The control program may instruct the instrument to modify its behavior to provide more control when control is needed.

[0057] Figure 3 shows an exemplary HCP monitoring system 20002 having a surgical hub 20006 paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain embodiments, as shown in Figure 3, the hub 20006 further includes a fume extraction module 20054 and / or a suction / irrigation module 20055. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, suction of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during surgery can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The hub modular enclosure 20060 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglement. Aspects of this disclosure present a surgical hub 20006 for use in surgical procedures involving the application of energy to tissue at the surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combination generator module slidably received within a docking station of the hub enclosure 20060. The docking station includes data and power contacts. The combination generator module includes two or more of the following components housed in a single unit: ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components.In one embodiment, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the fume exhaust component. In one embodiment, the fluid line may be a first fluid line, and a second fluid line may extend from the remote surgical site to a suction and irrigation module 20055 slidably received within a hub enclosure 20060. In one embodiment, the hub enclosure 20060 may include a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of the present disclosure present a solution in which a hub modular enclosure 20060 is configured to house different generators and facilitate bidirectional communication between them. One of the advantages of the hub modular enclosure 20060 is that it allows for the rapid removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the electrical engagement with the first power and data contacts.In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, distinct from a first energy, and a second docking station having a second docking port including a second data contact and a second power contact, wherein the second energy generator module is slidably movable to electrically engage with the power contact and the data contact, and the second energy generator module is slidably movable to disengage from the electrical engagement with the second power contact and the second data contact. In addition, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port, configured to facilitate communication between the first energy generator module and the second energy generator module. Referring to Figure 3, an aspect of the present disclosure relating to a hub modular enclosure 20060 enabling modular integration of a generator module 20050, a smoke extraction module 20054, and a suction / irrigation module 20055 is presented. The hub modular enclosure 20060 further facilitates bidirectional communication between modules 20059, 20054, and 20055. The generator module 20050 may be a generator module 20050 having integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the hub modular enclosure 20060. The generator module 20050 may be configured to connect to a unipolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the hub modular enclosure 20060. The hub modular enclosure 20060 can be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub modular enclosure 20060, so that multiple generators function as a single generator.

[0058] Figure 4 shows a surgical data network, according to at least one aspect of the present disclosure, having a set of communication hubs configured to connect a set of sensing systems, an environmental sensing system, and other modular devices to the cloud, located in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specifically equipped for surgical procedures.

[0059] As shown in Figure 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices located within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 which may include a remote server 20067 connected to remote storage 20068). The modular communication hub 20065 and devices may be connected in a room within the medical facility specifically equipped for surgical procedures. In one embodiment, the modular communication hub 20065 may also include a network hub 20061 and / or a network switch 20062 that communicates with a network router 20066. The modular communication hub 20065 may also be connected to a local computer system 20063 to provide local computer processing and data manipulation.

[0060] Computer system 20063 may comprise a processor and a network interface 20100. The processor may be connected via a system bus to communication modules, storage, memory, non-volatile memory, and input / output (I / O) interfaces. The system bus can be any of several types of bus structures, including a memory bus or memory controller, peripheral bus or external bus, and / or local bus, using any various available bus architectures. Examples of such architectures include, but are not limited to, a 9-bit bus, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0061] The processor may be any single-core or multi-core processor, such as those known by the trademark name ARM Cortex by Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0062] In one embodiment, the processor may include a safety controller comprising two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in order to provide highly integrated safety features while offering scalable performance, connectivity, and memory options.

[0063] It should be understood that computer system 20063 may include software that acts as an intermediary between the described user and basic computer resources in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0064] The user can input commands or information to the computer system 20063 via input devices connected to the I / O interface. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor 20102 via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to an output device. Output adapters may be provided to indicate that some output devices, such as monitors, displays, speakers, and printers, may be present, among others, which may require special adapters. Examples of output adapters include, but are not limited to, video and sound cards that provide means of connection between the output device and the system bus. Note that other devices and / or systems of devices, such as remote computers, may provide both input and output functions.

[0065] Computer system 20063 can operate in a networked environment using one or more remote computers, such as cloud computers, or logical connections to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described for computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated services digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0066] In various embodiments, the computer system 20063 may comprise an image processor, an image processing engine, a media processor, or any specialized digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction, multiple data (SIMD) or multiple-instruction, multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0067] The communication connection section may also refer to the hardware / software used to connect the network interface to the bus. For illustrative purposes, the communication connection section is shown as being inside the computer system 20063, but it may also be outside the computer system 20063. For illustrative purposes only, hardware / software required for connecting to the network interface may include internal and external technologies such as modems including typical telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0068] The surgical data network associated with the surgical hub system 20060 can be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to travel from one device (or segment) to another, and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional feature components that constitute each port within the network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0069] Modular devices 1a-1n, located in the operating room, may be connected to a modular communication hub 20065. A network hub 20061 and / or a network switch 20062 may be connected to a network router 20066 to connect devices 1a-1n to a cloud computing system 20064 or a local computer system 20063. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 20063 for local data processing and manipulation. Modular devices 2a-2m, located in the same operating room, may also be connected to a network switch 20062. The network switch 20062 may be connected to a network hub 20061 and / or a network router 20066 to connect devices 2a-2m to the cloud 20064. Data associated with devices 2a-2m may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. The data associated with devices 2a-2m may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0070] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing systems 20069 may include an HCP sensing system and / or a patient sensing system. One or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly via one of the network routers 20066, or via a network hub 20061 or network switch 20062 that communicates with the network router 20066.

[0071] The sensing system 20069 may be connected to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0072] As shown in Figure 4, the surgical hub system 20060 can be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. The modular communication hub 20065 can be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. The local computer system 20063 may also be housed in the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include a variety of modules, such as imaging modules connected to an endoscope, generator modules connected to energy-based surgical devices, fume extraction modules, suction / irrigation modules, communication modules, processor modules, storage arrays, surgical devices connected to displays, and / or non-contact sensor modules, among other modular devices that can be connected to the modular communication hub 20065 of the surgical data network.

[0073] In one embodiment, the surgical hub system 20060 shown in Figure 4 may comprise a combination of a network hub, network switch, and network router connecting devices 1a-1n / 2a-2m or sensing systems 20069 to a cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or sensing systems 20069 connected to the network hub 20061 or network switch 20062 may collect data or measurement data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” may be used as a metaphor for “Internet,” but the term is not limited to that. Therefore, the term “cloud computing” may be used herein to refer to “one type of internet-based computing,” in which various services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 20065 and / or computer system 20063 located in the operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. The cloud computing service may perform numerous calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0074] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, surgical data networks can provide improved surgical outcomes, reduced costs, and increased patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud computing system 20064 or the local computer system 20063, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be carried out for tissue-specific sites and conditions. Such data analysis may further utilize outcome analysis processing, and the use of standardized methods may provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0075] By applying cloud computing data processing techniques to measurement data collected by sensing systems 20069, surgical data networks can result in improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 may be used to assess the physiological state of a surgeon operating on a patient, a patient preparing for a surgical procedure, or a patient recovering after a surgical procedure. Cloud-based computing systems 20064 may be used to monitor biomarkers associated with a surgeon or patient in real time, generate surgical plans based at least on measurement data collected before a surgical procedure, supply control signals to surgical instruments during a surgical procedure, and notify the patient of complications during the postoperative period.

[0076] Operating room devices 1a-1n may be connected to the modular communication hub 20065 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub 20061. In one embodiment, the network hub 20061 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 20061 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 20061 also cannot store media access control (MAC / Internet Protocol, IP) for transferring device data. Only one of devices 1a-1n can transmit data through the network hub 20061 at a time. The network hub 20061 cannot have a routing table or intelligence regarding the destination of information and broadcasts all network data to each connection and to the remote server 20067 of the cloud computing system 20064. While the Network Hub 20061 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and may cause bottlenecks.

[0077] Operating room devices 2a-2m may be connected to network switch 20062 via a wired or wireless channel. Network switch 20062 operates within the data link layer of the OSI model. Network switch 20062 may also be a multicast device for connecting devices 2a-2m located in the same operating room to the network. Network switch 20062 transmits data in the form of frames to network router 20066 and may operate in full-duplex mode. Multiple devices 2a-2m may transmit data simultaneously through network switch 20062. Network switch 20062 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0078] Network hub 20061 and / or network switch 20062 may be coupled to network router 20066 to connect to cloud computing system 20064. Network router 20066 operates within the network layer of the OSI model. Network router 20066 creates routes for sending data packets received from network hub 20061 and / or network switch 20062 to cloud-based computing resources for further processing and manipulation of data collected by one or all of devices 1a-1n / 2a-2m and wearable sensing system 20011. Network router 20066 may be used to connect two or more different networks located in different locations, for example, different networks located in different operating rooms of the same medical facility, or different networks located in different operating rooms of different medical facilities. Network router 20066 transmits data in packet form to cloud computing system 20064 and may operate in full-duplex mode. Multiple devices can transmit data simultaneously. Network router 20066 may use IP addresses to transfer data.

[0079] In one embodiment, the network hub 20061 may be implemented as a USB hub that enables the connection of multiple USB devices to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 20061 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0080] In the example, the operating room devices 1a-1n / 2a-2m and / or sensing system 20069 may communicate with a modular communication hub 20065 via the Bluetooth wireless technology standard to exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and to build a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 may communicate with the modular communication hub 20065 via several wireless or wired communication standards or protocols, including but not limited to Bluetooth, Low-Energy Bluetooth, near-field communication (NFC), Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to short-range wireless communication such as Wi-Fi, Bluetooth, Low-Energy Bluetooth, and Bluetooth Smart, while the second communication module may be dedicated to long-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, and TDMA.

[0081] The modular communication hub 20065 functions as a central connection point for one or more of the operating room devices 1a-1n / 2a-2m and / or sensing systems 20069, and is capable of handling a data type known as a frame. A frame can carry data generated by the devices 1a-1n / 2a-2m and / or sensing systems 20069. When a frame is received by the modular communication hub 20065, it is amplified and transmitted to the network router 20066, which can then forward this data to the cloud computing system 20064 or the local computer system 20063 by using a number of wireless or wired communication standards or protocols as described herein.

[0082] The modular communication hub 20065 may be used as a standalone device or connected to compatible network hubs 20061 and network switches 20062 to form a larger network. Because the modular communication hub 20065 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room devices 1a-1n / 2a-2m.

[0083] Figure 5 shows a computer-implemented bidirectional surgical system 20070, which may be part of the HCP monitoring system 20002. The computer-implemented bidirectional surgical system 20070 is similar in many ways to the HCP sensing system 20002. For example, the computer-implemented bidirectional surgical system 20070 may include one or more surgical subsystems 20072 that are similar in many ways to the HCP monitoring system 20002. Each sub-surgical system 20072 may include at least one surgical hub 20076 that communicates with a cloud computing system 20064, which may include a remote server 20077 and remote storage 20078. In one embodiment, the computer-implemented bidirectional surgical system 20070 may include a modular control tower 20085 connected to multiple operating room devices, such as a sensing system 20001, intelligent surgical instruments, robots, and other computerized devices located in the operating room.

[0084] As shown in the embodiment of Figure 5, the modular control tower 20085 can be connected to an imaging module 20088 which can be connected to an endoscope 20087, a generator module 20090 which can be connected to an energy device 20089, a fume exhaust module 20091, a suction / irrigation module 20092, a communication module 20097, a processor module 20093, a storage array 20094, a smart device / instrument 20095 which can optionally be connected to displays 20086 and 20084, respectively, and a non-contact sensor module 20096. The non-contact sensor module 20096 can measure the dimensions of the operating room and generate a map of the operating room using ultrasonic, laser-type, and / or similar non-contact measuring devices. Other distance sensors can be used to determine the boundaries of the operating room. In U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” which is incorporated herein by reference in its entirety, an ultrasonic-based non-contact sensor module may scan an operating room by transmitting bursts of ultrasound and receiving echoes as the ultrasound bursts reflect off the outer walls of the operating room. The sensor module may be configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. A laser-based non-contact sensor module may scan an operating room, for example, by transmitting laser light pulses, receiving laser light pulses reflected off the outer walls of the operating room, comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0085] The modular control tower 20085 may also communicate with one or more sensing systems 20069 and environmental sensing systems 20015. Sensing systems 20069 may be connected to the modular control tower 20085 directly via a router or via a communication module 20097. Operating room devices may be connected to cloud computing resources and data storage via the modular control tower 20085. A robotic surgery hub 20082 may also be connected to the modular control tower 20085 and cloud computing resources. In particular, devices / instruments 20095 or 20084 and human interface systems 20080 may be connected to the modular control tower 20085 via wired or wireless communication standards or protocols as described herein. The human interface system 20080 may include a display subsystem and a notification subsystem. The modular control tower 20085 is connected to a hub display 20081 (e.g., a monitor, screen) and can display and overlay images received from the imaging module 20088, the device / instrument display 20086, and / or other human interface systems 20080. The hub display 20081 can also display data received from devices connected to the modular control tower 20085, along with the images and overlay images.

[0086] Figure 6 shows a logic diagram of a control system 20220 for a surgical instrument or surgical tool according to one or more embodiments of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may include surgical equipment that is on hand and specific to the procedure, such as an imaging device, a surgical stapler, an energy device, an endocutter device, etc. For example, the surgical instrument may include any of the following: a powered stapler, a powered stapler generator, an energy device, a high-energy device, a high-energy jaw device, an endocutter clamp, an energy device generator, an intraoperative imaging system, a fume ventilator, a suction irrigation device, a pneumoperitoneum system, etc. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221 having a processor 20222 and memory 20223. For example, one or more of the sensors 20225, 20226, and 20227 provide real-time feedback to the processor 20222. Motor 20230, driven by motor driver 20229, drives the I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to a processor 20222, which may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control I-beam firing, closure tube movement, shaft rotation, and joint movement. A display 20224 may display various operating states of the instrument and may include touchscreen functionality for data input. Information displayed on display 20224 may be overlaid with images acquired via the endoscopic imaging module.

[0087] The microcontroller 20221 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 20221 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance to above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs having 12 analog input channels.

[0088] The 20221 microcontroller may also include a safety controller comprising two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the Hercules ARM Cortex R4 trade names. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in particular, to provide advanced integrated safety features while offering scalable performance, connectivity, and memory options.

[0089] The microcontroller 20221 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 20221 may include a processor 20222 and memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 20229 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0090] The microcontroller 20221 can be programmed to provide precise control over the velocity and position of the displacement member and joint motion system. The microcontroller 20221 can be configured to calculate the response within its software. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response may be a suitable adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and this can detect external influences on the system.

[0091] Motor 20230 may be controlled by motor driver 20229 and utilized by a surgical instrument or tool launching system. In various forms, motor 20230 may be a brushed DC-driven motor having a maximum rotational speed of approximately 25,000 RPM. In some examples, motor 20230 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. Motor driver 20229 may comprise, for example, an H-bridge driver with field-effect transistors (FETs). Motor 20230 may be powered by a power supply assembly removably mounted to a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for supplying power to a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cell may be a lithium-ion battery that can be connected to and detached from a power supply assembly.

[0092] The motor driver 20229 may be the A3941, available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include a proprietary charge pump regulator that can supply full (>10V) gate drive to battery voltages down to 7V, allowing the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to supply the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side or low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be readily substituted for use in the 20228 tracking system with an absolute positioning system.

[0093] The tracking system 20228 may comprise a controlled motor drive circuit configuration comprising a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element which may be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 20225 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include contact-type or non-contact-type displacement sensors.A linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0094] The electric motor 20230 may include a rotary shaft operably connected to a set of drive teeth on a displacement member, or to a gear assembly mounted in a meshing engagement with a rack of drive teeth. A sensor element may be operably connected to a gear assembly such that one rotation of the position sensor element 20225 corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply provides power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0095] One rotation of the sensor element associated with the position sensor 20225 can correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor configuration may be connected via gear reduction, resulting in the position sensor 20225 completing one or more rotations over the full stroke of the displacement member. The position sensor 20225 may complete multiple rotations over the full stroke of the displacement member.

[0096] To provide a unique position signal for two or more rotations of the position sensor 20225, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches can be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is supplied to the microcontroller 20221. The sensor configuration of the position sensor 20225 may include an array of analog rotation sensors such as a magnetic sensor, a potentiometer, or an analog Hall effect element that outputs a unique combination of position signals or values.

[0097] The position sensor 20225 may comprise any number of magnetic sensing elements, such as magnetic sensors classified according to whether or not they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may encompass numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0098] The position sensor 20225 of the tracking system 20228, which includes an absolute positioning system, may include a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor, available from Austria Microsystems, AG. The position sensor 20225 is connected to a microcontroller 20221 to realize the absolute positioning system. The position sensor 20225 is a low-voltage, low-power component and may include four Hall effect elements in the area of ​​the position sensor 20225, which may be positioned above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement a simple and efficient algorithm for computing hyperbolic and trigonometric functions that require only addition, subtraction, bit shifts, and table lookup operations. Angular position, alarm bits, and magnetic field information can be transmitted to the microcontroller 20221 via a standard serial communication interface, such as a serial peripheral interface (SPI). The position sensor 20225 may offer 12-bit or 14-bit resolution. The position sensor 20225 may also be an AS5055 chip, available in a small QFN 16-pin 4×4×0.85mm package.

[0099] The tracking system 20228, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of the physical system will be by knowing the input, the calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, and inductive resistance.

[0100] The absolute positioning system can provide the absolute position of the displacement member when the device is powered on, without requiring a conventional rotary encoder to estimate the position of the device actuator, drive bar, knife, etc., which simply counts the number of steps taken forward or backward by the motor 20230, and does not require the displacement member to be moved back or forward to a reset (zero or home) position.

[0101] For example, a sensor 20226, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the closing force applied to the anvil and the amplitude of the strain exerted on the anvil during the clamping operation. The measured strain may be converted into a digital signal and provided to the processor 20222. Instead of, or in addition to, sensor 20226, a sensor 20227, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, sensor 20227, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and make deformable contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 20231 may be used to measure the current consumed by motor 20230. The force required to propel the launching member forward may, for example, correspond to the current consumed by motor 20230. The measured force can be converted into a digital signal and provided to processor 20222.

[0102] For example, a strain gauge sensor 20226 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include, for example, a strain gauge sensor 20226, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 20226 may measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and supplied to the processor 20222 of the microcontroller 20221. A load sensor 20227 may measure the force used to operate a knife element to cut tissue trapped between an anvil and a staple cartridge, for example. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor may also be converted into a digital signal and supplied to the processor 20222.

[0103] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 20226 and 20227 respectively, may be used by the microcontroller 20221 to characterize the selected position of the launching member and / or the corresponding values ​​of the launching member's velocity. In one embodiment, memory 20223 may store techniques, equations, and / or lookup tables that may be used by the microcontroller 20221 during evaluation.

[0104] The surgical instrument or tool control system 20220 may also include wired or wireless communication circuits for communicating with a modular communication hub 20065, as shown in Figure 5.

[0105] Figure 7 shows an exemplary sensing system 20069. The sensing system may be an HCP sensing system or a patient sensing system. Sensing system 20069 may include a sensor unit 20235 and a human interface system 20242 that communicate with a data processing and communication unit 20236. The data processing and communication unit 20236 may include an analog-to-digital converter 20237, a data processing unit 20238, a storage unit 20239, an input / output interface 20241, and a transceiver 20240. Sensing system 20069 can communicate with a surgical hub or computing device 20243, which then communicates with a cloud computing system 20244. The cloud computing system 20244 may include a cloud storage system 20078 and one or more cloud servers 20077.

[0106] The sensor unit 20235 may include one or more in vitro or in vivo sensors to measure one or more biomarkers. Biomarkers may include, for example, blood pH, hydration status, oxygen saturation, core body temperature, heart rate, heart rate variability, sweat rate, skin conductance, blood pressure, light tolerance, ambient temperature, respiratory rate, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, tissue perfusion pressure, bacteria in the airways, alcohol consumption, lactic acid (sweat), peripheral temperature, positive and optimistic moods, adrenaline (sweat), cortisol (sweat), edema, mycotoxins, VO2 max, preoperative pain, airborne chemicals, circulating tumor cells, stress and anxiety, confusion and delirium, physical activity, autonomic tension, circadian rhythm, menstrual cycle, and sleep. These biomarkers may be measured using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The sensors may measure the biomarkers as described herein using one or more sensing techniques such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary spectroscopy, potentiometric measurement, and current measurement.

[0107] As shown in Figure 7, the sensor in sensor unit 20235 may measure physiological signals (e.g., voltage, current, PPG signal, etc.) associated with the biomarker to be measured. The physiological signals to be measured may depend on the sensing technique used, as described herein. Sensor unit 20235 of sensing system 20069 may communicate with data processing and communication unit 20236. In one embodiment, sensor unit 20235 may communicate with data processing and communication unit 20236 using a wireless interface. Data processing and communication unit 20236 may include an analog-to-digital converter (ADC) 20237, a data processing unit 20238, storage 20239, an I / O interface 20241, and an RF transceiver 20240. Data processing unit 20238 may include a processor and a memory unit.

[0108] The sensor unit 20235 may transmit the measured physiological signal to the ADC 20237 of the data processing and communication unit 20236. In one embodiment, the measured physiological signal may pass through one or more filters (e.g., an RC low-pass filter) before being transmitted to the ADC. The ADC may convert the measured physiological signal into measurement data associated with the biomarker. The ADC may pass the measurement data to the data processing unit 20238 for processing. In one embodiment, the data processing unit 20238 may transmit the measurement data associated with the biomarker to a surgical hub or computing device 20243, and then to a cloud computing system 20244 for further processing. The data processing unit may transmit the measurement data to the surgical hub or computing device 20243 using one of the radio protocols as described herein. In one embodiment, the data processing unit 20238 may first process the raw measurement data received from the sensor unit and then transmit the processed measurement data to the surgical hub or computing device 20243.

[0109] In one embodiment, the data processing and communication unit 20236 of the sensing system 20069 may receive thresholds associated with biomarkers for monitoring from the surgical hub, computing device 20243, or directly from the cloud server 20077 of the cloud computing system 20244. The data processing unit 20236 may compare the measurement data associated with the monitored biomarker with the corresponding threshold received from the surgical hub, computing device 20243, or cloud server 20077. The data processing and communication unit 20236 may send a notification message to HID20242 indicating that the measurement data value has exceeded the threshold. The notification message may include the measurement data associated with the monitored biomarker. The data processing and computing unit 20236 may send the notification via transmission to the surgical hub or computing device 20243 using one of the RF protocols such as Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), or Wi-Fi. The data processing unit 20238 can send notifications (e.g., notifications about HCPs) directly to a cloud server via transmission to a cellular transmit / receive point (TRP) or base station, using one or more cellular protocols such as GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G. In one embodiment, the sensing unit may communicate with a hub / computing device via a router.

[0110] In one embodiment, the sensor unit may include a sensor and an analog-to-digital converter (ADC). The ADC in the sensor unit can convert the physiological signal measured by the sensor into measurement data associated with a biomarker. The sensor unit may transmit the measurement data to a data processing and communication unit for further processing. In one embodiment, the sensor unit may transmit the measurement data to the data processing and communication unit using an inter-integrated circuit (I2C) interface.

[0111] The data processing and communication unit includes a data processing unit, a storage unit, and an RF transceiver. The sensing system communicates with a surgical hub or computing device, which may then communicate with a cloud computing system 20244. The cloud computing system 20244 may include a remote server 20077 and associated remote storage 20078. The sensor unit may include one or more in vitro or in vivo sensors to measure one or more biomarkers as described herein.

[0112] The data processing and communication unit processes the measurement data received from the sensor unit, then processes the measurement data further, and / or transmits the measurement data to a smart hub or computing device 20243. In one embodiment, the data processing and communication unit may transmit the measurement data received from the sensor unit to a remote server 20077 of a cloud computing system 20244 for further processing and / or monitoring.

[0113] In one embodiment, the sensor unit may include a plurality of sensors for measuring one or more physiological signals associated with a patient or surgeon biomarker and / or one or more physical state signals associated with a person's physical state. The list of biomarkers may include biomarkers such as those disclosed herein. An ADC within the sensor unit may convert each of the physiological signals and / or physical state signals measured by the plurality of sensors into corresponding measurement data. The sensor unit may transmit the measurement data associated with one or more biomarkers, and with the physical state of the monitored patient, to a data processing and communication unit for further processing. The sensor unit may transmit the measurement data to the data processing and communication unit individually for each sensor or combined for all sensors. In one embodiment, the sensor unit may transmit the measurement data to the data processing and communication unit via an I2C interface.

[0114] Figure 8 shows an example of using surgical task situation recognition and measurement data from one or more HCP sensing systems to coordinate surgical instrument control. Figure 8 shows an exemplary surgical procedure timeline 20265 and contextual information that the surgical hub may derive from data received from one or more surgical devices, one or more HCP sensing systems, and / or one or more environmental sensing systems at each step of the surgical procedure. Devices that can be controlled by the surgical hub may include high energy devices, endocutter clamps, etc. The HCP sensing system may include sensing systems for measuring one or more biomarkers associated with the surgeon, e.g., heart rate, sweat composition, respiratory rate, etc. The environmental sensing system may include systems for measuring one or more environmental attributes, e.g., cameras for detecting the surgeon's position / movement / breathing patterns, spatial microphones for measuring ambient noise and / or the tone of the healthcare provider's voice in the operating room, ambient temperature / humidity, etc.

[0115] In the following description of the timeline 20265 shown in Figure 8, please also refer to Figure 5. Figure 5 provides various components used in a surgical procedure. Timeline 20265 shows steps that may be performed individually and / or collectively by nurses, surgeons, and other healthcare professionals during an exemplary colorectal surgery. In a colorectal surgery, the situation-aware surgical hub 20076 may receive data from various data sources throughout the surgical procedure, including data generated each time the HCP utilizes the modular device / instrument 20095 paired with the surgical hub 20076. The surgical hub may receive this data from the paired modular device 20095. The surgical hub may receive measured data from the sensing system 20069. The surgical hub may continuously derive the surgeon's stress level and inferences (i.e., contextual information) regarding the procedure in progress as new data is received, using data from modular devices / instruments 20095 and / or measurement data from sensing systems 20069, so that the surgeon's stress level regarding the procedure being performed is obtained. The situation awareness system of the surgical hub 20076 may perform one or more of the following: record data regarding the procedure to generate a report; verify the steps being performed by the healthcare professional; provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen); adjust modular devices based on the situation (e.g., activate a monitor, adjust the FOV of a medical imaging device, change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument); or perform any other such actions described herein. In one embodiment, these steps may be performed by a remote server 20077 of a cloud system 20064, which may communicate with the surgical hub 20076.

[0116] As a first step (not shown in Figure 8 for brevity), a hospital staff member may search the hospital's EMR database for the patient's EMR. Based on the patient data selected in the EMR, the surgical hub 20076 may determine that the procedure to be performed is a thoracic procedure. The staff member may scan the medical supplies coming in for the procedure. The surgical hub 20076 may cross-reference the scanned supplies with a list of supplies that can be used in various types of procedures and confirm that the combination of supplies matches a thoracic procedure. The surgical hub 20076 may pair each of the sensing systems 20069 fitted by different HCPs.

[0117] Once each device is prepared and preoperative preparations are complete, the surgical team may begin by making an incision and positioning the trocar. The surgical team may perform access and preparation by incising any adhesions and identifying the inferior mesenteric artery (IMA) branch. The surgical hub 20076 may infer that the surgeon is in the process of incising adhesions, at least based on data that can be received from an RF or ultrasound generator indicating that an energy instrument is being emitted. The surgical hub 20076 may cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (e.g., after the completion of the steps of the procedure described above) corresponds to the incision step.

[0118] Following dissection, the HCP may proceed to the ligation step of the procedure (e.g., indicated by A1). As shown in Figure 8, the HCP may begin by ligating the IMA. The surgical hub 20076 may receive data from the high-energy jaw device and / or endocutter indicating that the instrument is firing, thus allowing it to infer that the surgeon is ligating the arteries and veins. The surgical hub may also receive measured data from one of the HCP's sensing systems indicating a high stress level in the HCP (e.g., indicated by the B1 mark on the time axis). For example, a high stress level may be indicated by a change in the HCP's heart rate from a baseline. The surgical hub 20076 may derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps retrieved in the process (e.g., as shown by A2 and A3), as in the previous step. During the high-stress period, the surgical hub 20076 may monitor the high-energy jaw trigger ratio, as well as the endocutter clamp and firing rate. In one embodiment, the surgical hub 20076 may transmit auxiliary control signals to a high-energy jaw device and / or an end-cutter device to control the device in operation. The surgical hub may transmit auxiliary signals based on the stress level of the HCP operating the surgical device and / or the situational awareness known to the surgical hub. For example, the surgical hub 20076 may transmit control auxiliary signals to the high-energy device or end-cutter clamp as shown by A2 and A3 in Figure 8.

[0119] The HCP may proceed to the next step of releasing the upper sigmoid colon, followed by the descending colon, rectum, and sigmoid colon. The surgical hub 20076 may continue to monitor the HCP's high-stress markers (e.g., as indicated by D1, E1a, E1b, F1). During periods of high stress, the surgical hub 20076 may transmit auxiliary signals to high-energy jaw devices and / or endocutter devices, as shown in Figure 8.

[0120] After mobilizing the colon, the HCP may proceed to the segmental resection portion of the procedure. For example, the surgical hub 20076, based on data from surgical stapling and cutting instruments, including data from its cartridge, can infer that the HCP is transversely incising the intestine and removing the sigmoid colon. Cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Since different types of staples are used for different types of tissue, cartridge data may indicate the type of tissue being stapled and / or transversely incised. Note that, since various instruments are well suited to specific tasks, surgeons should periodically switch between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the stage of the procedure. Therefore, the sequence in which stapling / cutting instruments and surgical energy instruments are used may indicate which stage of the procedure the surgeon is performing.

[0121] The surgical hub may determine and transmit control signals to surgical devices based on the stress level of the HCP. For example, during period G1b, control signal G2b may be transmitted to the endocutter clamp. Once the sigmoid colon is removed, the incision may be closed and the postoperative portion of the procedure may begin. The patient may be awakened from anesthesia. The surgical hub 20076 may infer that the patient is awakened from anesthesia based on one or more sensing systems attached to the patient.

[0122] Figure 9 shows an exemplary computer-implemented bidirectional surgical system that may be configured to monitor HCP biomarkers using one or more sensing systems 20001. The computer-implemented bidirectional surgical system may be configured to monitor HCP biomarkers using one or more sensing systems 20069. HCP biomarkers and / or patient biomarkers may be measured before, after, and / or during surgical procedures. In one embodiment, the computer-implemented bidirectional surgical system may be configured to monitor and analyze data on the operation of various surgical systems 20069, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented bidirectional surgical system may include a cloud-based analysis system. The computer-implemented bidirectional surgical system may include a local analysis system. The cloud-based analysis system may include one or more analysis servers.

[0123] The monitoring and analysis system may include multiple sensing systems 20268 (which may be the same as or similar to sensing system 20069), surgical instruments 20266 (which may be the same as or similar to instrument 20031), and / or multiple surgical hubs 20270 (which may be the same as or similar to hub 20006). A surgical data network 20269 (which may be the same as or similar to the surgical data network shown in Figure 4) may connect the surgical hubs 20270 to a computing system 20271 (which may be a local computing system, an edge computing system, or a cloud computing system such as cloud computing system 20064). The surgical hubs 20270 may be communicatively connected to one or more surgical instruments 20266. The surgical hubs 20270 may also be communicatively connected to one or more sensing systems 20268 and to a cloud of computer-implemented bidirectional surgical systems 20271 via the network 20269. The surgical hub 20270 and the sensing system 20268 may be connected in a communicative manner using wireless protocols as described herein. The computing system 20271 may be a local or remote focus source of hardware and software for storing, processing, manipulating, and communicating measurement data from the sensing system 20268 and data generated based on the operation of various surgical instruments 20266.

[0124] As shown in Figure 9, access to the computing system 20271 can be achieved via network 20269. Network 20269 may be the Internet or any other suitable computer network. A surgical hub 20270, which can be connected to the computing system 20271, can be considered the client side of the computing system (e.g., a cloud-based analytics system). Surgical instruments 20266 may be paired with the surgical hub 20270 for the control and implementation of various surgical procedures and / or operations, as described herein. A sensing system 20268 may be paired with the surgical hub 20270 for intraoperative, preoperative, or postoperative HCP monitoring of surgeon-related biomarkers. An environmental sensing system 20267 may be paired with the surgical hub 20270 to measure environmental attributes associated with HCP.

[0125] Surgical instruments 20266, environmental sensing systems 20267, and sensing systems 20268 may be equipped with wired or wireless transceivers for data transmission to and from their corresponding surgical hubs 20270 (which may also be equipped with transceivers). One or more combinations of surgical instruments 20266, sensing systems 20268, or surgical hubs 20270 may indicate specific locations within a medical facility (e.g., a hospital), such as operating rooms, intensive care units (ICUs), or recovery rooms, for the purpose of providing medical procedures, pre-operative preparations, and / or post-operative recovery. For example, the memory of a surgical hub 20270 may store location data.

[0126] As shown in Figure 9, the cloud system 20271 may include one or more central servers 20272 (which may be the same as or similar to remote servers 20067), a surgical hub application server 20276, a data analysis module 20277, and an input / output ("I / O") interface 20278. The central servers 20272 of the computing system 20271 can collectively manage the cloud computing system, including monitoring requests from client surgical hubs 20270 and managing the processing power of the computing system 20271 to perform those requests. Each of the central servers 20272 may comprise one or more processors 20273 coupled to a preferred memory device 20274, which may include volatile memory such as random-access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 20274 is equipped with machine-executable instructions, which, when executed, can cause the processor 20273 to execute the data analysis module 20277 for cloud-based data analysis, real-time monitoring of measurement data received from the sensing system 20268, and for operations, recommendations, and other operations as described herein. The processor 20273 may execute the data analysis module 20277 independently or in conjunction with a hub application that runs independently by the hub 20270. The central server 20272 may also include a database 20275 of aggregated medical data that may reside in memory 20274.

[0127] Based on connections to various surgical hubs 20270 via network 20269, computing system 20271 can aggregate specific data generated by various surgical instruments 20266, real-time data from sensing systems 20268, and / or data from surgical hubs 20270. Such aggregated data may be stored in an aggregated medical database 20275 associated with computing system 20271. Computing system 20271 can track real-time measurement data from sensing systems 20268 and / or perform data analysis and actions on the measurement data and / or aggregated data to gain insights and / or perform functions that individual hubs 20270 could not achieve on their own.

[0128] As shown in Figure 9, the computing system 20271 and the surgical hub 20270 are connected in a communicative manner to send and receive information. The I / O interface 20278 connects to multiple surgical hubs 20270 via the network 20269. The I / O interface 20278 may be configured to transfer information between the surgical hubs 20270 and the aggregated medical data database 20275. The I / O interface 20278 may facilitate read / write operations of a cloud-based analysis system. Such read / write operations may be performed in response to requests from the hubs 20270. These requests may be sent to the surgical hubs 20270 via the hub application. The I / O interface 20278 may include one or more high-speed data ports, including a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the computing system 20271 to the surgical hubs 20270. The hub application server 20276 of computing system 20271 may be configured to host and supply shared functions to a software application (e.g., a hub application) running on the surgical hub 20270. For example, the hub application server 20276 may manage requests made through the hub 20270 by the hub application, control access to the aggregated medical data database 20275, and perform load balancing.

[0129] The computing system can address various issues arising in the context of medical procedures (e.g., preoperative, intraoperative, and postoperative monitoring) and treatments performed using medical devices such as surgical instruments 20266 and 20031. Surgical instrument 20266 may be a digital surgical device configured to interact with cloud 20271 to implement techniques for improving surgical performance. The computing system can address various issues arising in the context of monitoring one or more biomarkers associated with HCP or the patient during preoperative, intraoperative, and postoperative procedures using sensing system 20268. Sensing system 20268 may interact with surgical hub 20270 and / or computing system 20271 to implement techniques for monitoring surgeon biomarkers and / or patient biomarkers. The sensing system 20268 may be a system having one or more sensors configured to measure one or more biomarkers associated with an HCP participating in a medical procedure and / or a patient on whom a medical procedure is scheduled, is being performed, or has been performed. Various surgical instruments 20266, sensing system 20268, and / or surgical hub 20270 may include a human interface system (e.g., having a touch-controlled user interface) so that the HCPS and / or patient can control the manner of interaction between the surgical instrument 20266 or sensing system 20268 and the computing system 20271. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0130] Figure 10 shows an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 that can communicate with a console 20294 or a portable device 20296 via a local area network 20292 and / or a cloud network 20293 via a wired or wireless connection. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 so that the adapter 20285 transmits force from the drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) located inside for measuring the force acting on the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may also be a bio-implant, i.e., a multi-firing loading unit (MFLU), which allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 20287 from the surgical site in order to reload the loading unit 20287.

[0131] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that can be fired two or more times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener as it is ejected from the multi-fire fastener cartridge.

[0132] The handle 20297 may include a motor connected to the drive shaft so as to act on the rotation of the drive shaft. The handle 20297 may include a control interface for selectively starting the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces, which can be engaged by a clinician to start the motor.

[0133] The control interface of the handle 20297 may communicate with the controller 20298 of the handle 20297 to selectively start the motor and act on the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and may be configured to receive input from the control interface and adapter data from adapter 20285 or loading unit data from loading unit 20287. The controller 20298 may analyze the input from the control interface and the data received from adapter 20285 and / or loading unit 20287 in order to selectively start the motor. The handle 20297 may also include a display that can be viewed by a clinician while the handle 20297 is in use. The display may be configured to show portions of the adapter or loading unit data before, during, or after firing the instrument 20282.

[0134] Adapter 20285 may include an internally located adapter identification device 20284, and loading unit 20287 may include an internally located loading unit identification device 20288. Adapter identification device 20284 communicates with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It will be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, which relays or passes through communication from loading unit identification device 20288 to controller 20298.

[0135] Adapter 20285 may also include a plurality of sensors 20286 (one shown) positioned around it to detect various states of Adapter 20285 or the environment (e.g., whether Adapter 20285 is connected to a loading unit, whether Adapter 20285 is connected to a handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside Adapter 20285, the number of times Adapter 20285 has fired, the peak force of Adapter 20285 during firing, the total amount of force applied to Adapter 20285, the peak recoil force of Adapter 20285, the number of pauses of Adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to Adapter Identification Device 20284 in the form of data signals. The data signals from the plurality of sensors 20286 may be stored in Adapter Identification Device 20284 or used to update Adapter data stored in Adapter Identification Device 20284. The data signals from the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges for measuring the force exerted on the loading unit 20287 during firing.

[0136] The handle 20297 and adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). Additionally or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductively). It is also intended that the adapter identification device 20284 and the controller 20298 may wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0137] Handle 20297 may include transceiver 20283 configured to transmit instrument data from controller 20298 to other components of system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). Controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to surgical hub 20270, as illustrated in Figure 9. Transceiver 20283 may receive data from surgical hub 20270 (e.g., cartridge data, loading unit data, adapter data, or other notifications). Transceiver 20283 may receive data from other components of system 20280 (e.g., cartridge data, loading unit data, or adapter data). For example, controller 20298 may transmit instrument data to console 20294, including the serial number of a mounting adapter (e.g., adapter 20285) attached to handle 20297, the serial number of a loading unit (e.g., loading unit 20287) attached to adapter 20285, and the serial number of a multi-shot fastener cartridge loaded into the loading unit. Console 20294 may then send data associated with the mounted cartridge, loading unit, and adapter, respectively (e.g., cartridge data, loading unit data, or adapter data), back to controller 20298. Controller 20298 may display the message on the local instrument display, or send the message via transceiver 20283 to console 20294 or portable device 20296, which will then display the message on display 20295 or the portable device screen, respectively.

[0138] Figure 11 shows a diagram of a situation-aware surgical system 5100 according to at least one aspect of the present disclosure. The data source 5126 may include, for example, a modular device 5102 (which may include sensors configured to detect parameters associated with the patient, HCP, environment and / or the modular device itself), a database 5122 (e.g., an EMR database including patient records), and patient monitoring devices 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor), an HCP monitoring device 35510, and / or an environment monitoring device 35512. The surgical hub 5104 may be configured to derive contextual information about a surgical procedure from the data, for example, based on a specific combination of received data or a specific order in which the data was received from the data source 5126. Contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a specific step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function of the surgical hub 5104 for deriving or inferring information about a surgical procedure from received data may be referred to as “situational awareness.” For example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104, for deriving contextual information related to a surgical procedure from received data.

[0139] The situation awareness system of the surgical hub 5104 can be configured to derive contextual information from data received from the data source 5126 in various different ways. For example, the situation awareness system may include a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 5122, patient monitoring device 5124, modular device 5102, HCP monitoring device 35510, and / or environmental monitoring device 35512) with corresponding contextual information about the surgical procedure. The machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In the example, the situation awareness system may include a lookup table that stores pre-characterized contextual information about the surgical procedure, associated with one or more inputs (or ranges of inputs) that correspond to that contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situation awareness system to control the modular device 5102. In the example, contextual information received by the situation awareness system of the surgical hub 5104 can be associated with a specific control adjustment of one or more modular devices 5102, or a set of control adjustments. In the example, the situation awareness system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments of one or more modular devices 5102 when given contextual information as input.

[0140] The surgical hub 5104, which incorporates a situational awareness system, can offer many advantages to the surgical system 5100. One advantage may include providing improved interpretation of the sensed and collected data, which improves processing accuracy during the course of the surgical procedure and / or the use of the data. Returning to the previous example, the situational awareness surgical hub 5104 can determine what type of tissue is being operated on, and therefore, if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the tissue type.

[0141] The type of tissue being operated on can affect the adjustments made to the compression rate and load threshold of surgical staple and cutting instruments for measuring specific interstitial gaps. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic or abdominal procedure, thereby enabling the surgical hub 5104 to determine whether the tissue clamped by the end effector of the surgical staple and cutting instrument is lung tissue (in the case of a thoracic procedure) or gastric tissue (in the case of an abdominal procedure). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical staple and cutting instrument to match the type of tissue.

[0142] The type of body cavity being operated on during a blown procedure can affect the function of the fume extractor. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing air insufflation) and determine the type of procedure. Generally, since certain procedure types may be performed in specific body cavities, the surgical hub 5104 can appropriately control the motor speed of the fume extractor to match the body cavity being operated on. Thus, the situational awareness surgical hub 5104 can provide a consistent amount of fume extractor for both thoracic and abdominal procedures.

[0143] The type of procedure being performed can affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopy, the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid, which may require a higher energy level. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopy. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (e.g., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situational awareness surgical hub 5104 can determine the type of surgical procedure being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue shape for the surgical procedure. Furthermore, the situation-aware surgical hub 5104 can be configured to adjust the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument not simply for each procedure, but throughout the course of the surgical procedure. The situation-aware surgical hub 5104 can determine which step of the surgical procedure is being performed or will be performed, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy levels to values ​​appropriate for the expected tissue type according to the steps of the surgical procedure.

[0144] In the example, the surgical hub 5104 can derive data from an additional data source 5126 to improve conclusions drawn from one data source 5126. The context-aware surgical hub 5104 can enhance data received from the modular device 5102 with contextual information constructed from other data sources 5126 regarding the surgical procedure. For example, the context-aware surgical hub 5104 can be configured to determine whether hemostasis has occurred (e.g., whether bleeding at the surgical site has stopped) based on video or image data received from a medical imaging device. The surgical hub 5104 can be further configured to make a determination regarding the integrity of a staple line or tissue weld by comparing physiological measurements (e.g., blood pressure sensed by a BP monitoring device communicably connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device communicably connected to the surgical hub 5104). The context-aware system of the surgical hub 5104 can take physiological measurement data into consideration to provide additional context when analyzing visualization data. Additional context can be useful when the visualized data itself may not be definitive or may be incomplete.

[0145] For example, the situational awareness surgical hub 5104 may proactively start the generator to which the RF electrosurgical instrument is connected if it determines that the instrument will be needed in a subsequent step of the procedure. Proactively starting the energy source makes it possible to have the instrument ready for use as soon as the preceding step of the procedure is completed.

[0146] The situational awareness surgical hub 5104 can determine whether the current or subsequent steps of a surgical procedure require different views or magnifications on the display, according to the features of the surgical site that the surgeon is expected to need to see. The surgical hub 5104 can proactively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108) as needed, thereby automatically adjusting the display throughout the surgical procedure.

[0147] The situational awareness surgical hub 5104 can determine which steps of a surgical procedure are being performed or will be performed next, and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0148] Errors can be checked during the setup of a surgical procedure or during the procedure itself. For example, the situation-aware surgical hub 5104 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 can compare a list of items for the procedure and / or a list of devices paired with the surgical hub 5104 to a recommended or expected manifest of items and / or devices for a given surgical procedure. If any discontinuities exist between the lists, the surgical hub 5104 can provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, HCP monitoring device 35510, environmental monitoring device 35512, and / or other surgical items are missing. In some embodiments, the surgical hub 5104 can determine the relative distance or relative position of a modular device 5102 and a patient monitoring device 5124, for example, via a proximity sensor. The surgical hub 5104 can compare the relative positions of the devices to a recommended or expected layout for a particular surgical procedure. If any discontinuity exists between the layouts, the surgical hub 5104 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the proposed layout.

[0149] The situational awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making an error or deviating from a set of expected actions during a surgical procedure. For example, the surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. The surgical hub 5104 can provide alerts indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0150] Surgical instruments (and other modular devices 5102) can be adapted to the specific context of each surgical procedure (e.g., adaptation to different tissue types) and can verify actions during the surgical procedure. Subsequent steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the unique context of the procedure.

[0151] Object detection can be performed through computer vision and image processing, dealing with the detection of instances of specific classes of semantic objects (such as humans, machines, or objects) in digital images and videos. Object detection is used in computer vision tasks such as image annotation, activity recognition, face detection, face recognition, and video object co-segmentation. Object detection is used for tracking objects, e.g., tracking a ball during a football match, tracking the movement of a cricket bat, or tracking a person in a video. Object detection can be performed through neural network-based and / or non-neural methods. For example, feature parts may be defined, and classification may be performed based on these defined feature parts, for example, via a support vector machine (SVM). Neural techniques can perform object detection without specifically defining feature parts and may be based on convolutional neural networks (CNNs).

[0152] Moving object detection can be performed to recognize the physical movement of people or objects in a given location or area. By segmenting between moving objects and stationary areas or regions, the movement of moving objects can be tracked and therefore analyzed later. Moving object detection can be performed through background subtraction, frame subtraction, time subtraction, and / or optical flow analysis.

[0153] Smart and self-identifying RF systems can be used to locate HCPs and equipment within the OR. Spatial identification sensors can be used to locate HCPs, instruments, equipment, and / or boundaries. The locations of HCPs, equipment, and / or boundaries can be tracked to rooms and / or patients via wireless sensors and beacons.

[0154] Wearable monitoring systems can be linked to individuals such as HCPs or patients. Ultra-broadband monitoring can be used to identify a person's location within an OR (Operational Research) and / or their relative position to equipment, surgical instruments, other people, boundaries, etc.

[0155] RF element identification and boundary monitoring can be performed. RF beacons may be placed on equipment or devices. BLE beacons may be placed within a physical OR. For example, beacons may be placed at the corners of an OR that outline the boundaries of a room. BLE beacons may be placed on structures within an OR, such as large structures. BLE beacons may be placed on a device to monitor its location relative to one or more boundaries.

[0156] For example, RFID tags may be attached to equipment and / or HCPs. Electromagnetic gates may be placed within the OR to prevent or minimize boundary breaches.

[0157] Boundary violations can be detected by tracking the movement of beacons. For example, a surgical hub may detect a boundary violation if it determines that a beacon has moved across the boundary. If the boundary violation is associated with a device crossing the boundary, the surgical hub may shut down the device and alert the HCP of the change in status. If the boundary violation is a larger structure, an emergency message may be displayed to indicate that the equipment is of concern.

[0158] RF beacons can be used to locate HCPS locations within the OR theater. HCP locations can be tracked via their identification badges, which may have embedded RF beacons. RF beacons can be high frequency (HF) or ultra-high frequency (UHF). The reading range may not be affected by the need to penetrate fluids or structures. Because OR sizes can vary, HF or UHF may provide sufficient coverage to accommodate potential distances from the surgical hub.

[0159] Surgical instruments may be associated with unique RFID identification. The location of surgical instruments may be tracked. The current use and / or previous history of surgical instruments may be determined based on location tracking. External readers from the surgical hub may be positioned around the patient. External readers may be used to track surgical instruments entering the OR field. Low-frequency or HF chips may be used on the instruments. Low-frequency or high-frequency may allow the reader to penetrate bodily fluids.

[0160] Instrument use can be determined and tracked based on its location. For example, the number of times a surgical instrument has been used, the length of time the device was in the patient, and / or the level of security it underwent through a sterilization cycle before being introduced into the OR can be determined through location tracking. Whether an instrument was left in the patient can be detected through instrument location tracking.

[0161] OR room imaging may be performed by multiple sources through various techniques to determine the location and configuration of HCPs, instruments, and / or equipment. Two or more imaging devices through different energy types may be used to locate objects and people within the OR, OR walls, and / or their vicinity. Location and / or proximity information may be used to identify and / or control their interactions. Instruments and / or equipment may be configured based on the identified interactions.

[0162] For example, ultrasonic echolocation can be used to determine the movement and location of an object. OR vision or multispectral imaging can be used to monitor a user's location, interactions, and communications. Audio monitoring can be used to determine and record noise generation monitoring and tracking.

[0163] Infrared (IR) thermography monitoring may be used to monitor changes in core temperature as well as the temperature of the HCP and the patient, in order to adjust the system, monitor infection, and / or understand the situation. When the HCP enters the OR, an IR signature may be created for the HCP. The initial IR association response may be logged, for example, via a surgical hub. If the HCP's stress level increases during the procedure, the IR signature may display or register the increased stress level.

[0164] Various imaging systems may be used simultaneously. For example, a surgical hub may be positioned within the OR (Operational Regulatory Area). Once powered on, the surgical hub can use ultrasound echoes to determine the location of large instruments for mapping the OR boundaries (e.g., walls) and rooms. A grid map may be established using all necessary boundaries.

[0165] When an HCP enters the OR theater, the HCP can be identified as entering via a BLE beacon associated with the HCP. The surgical hub can inspect the HCP entering the theater. The surgical hub can initiate tracking of the HCP via an optical camera. An IR camera can perform an initial scan of the identified HCP for baseline stress levels. Once the procedure is initiated, the audio monitoring system can monitor and record sounds. The surgical hub can monitor vocal intonation and / or elevated nervous voice. The audio monitoring system can be synchronized with the IR camera, for example, to verify an increase in stress levels. This process can be performed for multiple HCPs.

[0166] HCPs within the OR can be recognized and tracked via ID badges having RFID, NFC, and / or wearable devices. For example, an HCP's identity can be determined when an HCP enters or leaves a room, when an HCP approaches another wearable device, or when an HCP approaches a surgical hub. When the surgical hub detects that an HCP has entered the OR, it may request HCP identification information from an RFID reader, NFC reader, or wearable device. The surgical hub may verify the HCP's identity based on other information sources. The surgical hub may assume the identity of an HCP associated with an ID badge or wearable device until the ID badge or wearable device is turned off. Further details relating to identifying HCPs and / or user roles via wearable devices can be found in U.S. Patent Application No. 17 / 156,324, entitled "ACTIVE RECOGNITION AND PAIRING SENSING SYSTEMS," which is incorporated herein by reference in its entirety.

[0167] HCPs within the OR can be recognized and tracked via video processing from video captured by one or more cameras within the OR. The video feed may be analyzed using a variety of known image or video processing techniques, such as keypoint detection and analysis, bounding box annotation, polygon meshing, image segmentation, face recognition, gesture recognition, point clouds, lines, and splines.

[0168] The location, movement, and / or orientation of various surgical products and instruments within the operating room can be identified and tracked. For example, a gyroscope or a three-axis accelerometer may be used to determine the orientation and position of a device.

[0169] For example, a surgical instrument may be identified via one or more spatial alignment markers positioned on it. For example, a visible reference marker may be placed on the instrument. The surgical hub may monitor the location, movement, and / or orientation of the surgical instrument via one or more cameras within the OR. The reference marks may be predefined patterns. The surgical hub may associate a surgical instrument with a specific reference mark and use that mark to identify and model the instrument in a 3D computer environment that it creates and records. Alignment allows for compensation of translation, rotation, scale, skew, and perspective. This enables the surgical hub to detect and monitor the instrument even if a portion of the instrument is obscured.

[0170] For visible monitoring, camera calibration may be performed when the system starts up. A predefined set of calibration markers may be fixed within the hub camera view so that the surgical hub can calibrate the camera for distance and focal length. The surgical hub may determine the exact distance from itself to another calibration preset scale in the OR, and use the measurement and scale to calibrate the camera and focal length.

[0171] For example, a surgical hub may determine precise distances via a measurement system. For instance, a measurement system utilizing laser Doppler, ultrasonic pinging, RF, and / or other energy digital communication may perform distance measurements and transmit the measurements to the surgical hub. The measurement system may be included within the surgical hub.

[0172] Distance can be inferred from active and / or passive electronic signal processing. For example, the distance between two paired systems or devices, and the distance between two systems or devices, can be determined by monitoring signal strength and compensating for emitted power, emitted device antenna path, flight path, receiving device antenna path, and / or receiver sensitivity. For example, UHF or HF RFID-tagged objects can be tracked through predefined tag-and-distance combinations in combination with unknown tags. Tags can be used to identify products, devices, or equipment, and once identified, they can enable tracking within the OR. Tags can provide further information about the identified product, device, or equipment. RFID maps can be generated from passive or active reference tags with known locations (e.g., landmarks) to pinpoint the location of any unknown tags detected by an RFID reader antenna. The distance between a reader and a common detected tag can be measured using a large-scale path loss propagation model. The distance between an unknown tag and a detected landmark (e.g., inter-tag distance) can be calculated.

[0173] Millimeter-wave radar can be used to track objects. Millimeter-wave radar can achieve an accuracy of a few micrometers. In radars that operate using frequency-modulated continuous waves (FMCW), the frequency and / or phase of the radar beat signal can be used to determine the distance between the radar sensor and the object to which the radar signal is reflected.

[0174] The boundaries of the operating room may be mapped or evaluated. For example, the surgical hub 20006 may maintain spatial awareness in operation by periodically mapping its operating room, which can be useful in determining whether the surgical hub 20006 has been moved. Re-evaluation may be performed periodically or triggered by events such as observing a change in devices of the HCP monitoring system 20002 that are considered to be in the operating room. The change may be the detection of a new device that was not previously considered to be within the boundaries of the operating room. The change may be the disappearance, disconnection, or unpairing of a paired device that was previously considered to reside in the operating room. The surgical hub 20006 may detect the disappearance, disconnection, or unpairing of a paired device by continuously monitoring its connection to the paired device.

[0175] The operating room mapping module may include a compass and an integrated Bluetooth transceiver. Other communication mechanisms not significantly affected by the hospital environment or geographical location may be used. Bluetooth Low Energy (BLE) beacon technology can achieve indoor distance measurement with an accuracy of approximately 1-2 meters and has improved accuracy at closer ranges (within 0-6 meters). A compass is used in conjunction with BLE to improve the accuracy of distance measurement. The operating room mapping module may use BLE and / or a compass to determine where the module is located relative to the patient. For example, two modules facing each other (detected by a compass) can clearly indicate that the modules are on either side of the patient when the distance between them is more than 1 meter. The presence of more "hub"-enabled modules in the operating room leads to higher achievable accuracy due to triangulation techniques. The operating room mapping module may be contained within the surgical hub 20006 as described herein. The operating room mapping module may communicate operably with the surgical hub 20006 as described herein.

[0176] The operating room mapping module can map the physical locations of devices and / or surgical modules that reside within the operating room. This information can be used by the user interface to display a virtual map of the operating room, allowing the user to more easily identify which modules are present and enabled, and their current status. Mapping data collected by the surgical hub 20006 can be analyzed, for example, to identify how the operating room is physically set up.

[0177] For example, the surgical hub 20006 can determine the location of a device by evaluating the transmitted radio signal strength and direction. For the Bluetooth protocol, the Received Signal Strength Indication (RSSI) is a measurement of the received radio signal strength. In one embodiment, the device of the HCP monitoring system 20002 may be equipped with a USB Bluetooth dongle. The surgical hub 20006 can obtain distance information by scanning USB Bluetooth beacons. For example, multiple high-gain antennas on a Bluetooth access point with a variable attenuator may produce more accurate results than RSSI measurements. The surgical hub 20006 can determine the location of a device by measuring the signal strength from multiple antennas.

[0178] The surgical hub 20006 can identify components of the HCP monitoring system 20002 once they are brought into the operating room. For example, devices of the HCP monitoring system 20002 may have identifiers recognizable by the surgical hub 20006, such as barcodes or RFID tags. NFC may also be used. The surgical hub 20006 may include a suitable reader or scanner for detecting devices brought into the operating room. [Further details relating to spatial recognition of surgical hubs in operating rooms can be found in U.S. Patent Application No. 15 / 940,666, filed March 29, 2018, entitled "SPATIAL AWARENESS OF SURGICAL HUBS IN OPERATING ROOMS," which is incorporated herein by reference in its entirety.]

[0179] The computing system may be or include an HCP monitoring system such as HCP monitoring systems 20000, 20002, 20003, or 20004 as described herein with respect to Figures 1 to 3. The computing system may be a computing system operably connected to HCP monitoring systems 20000, 20002, 20003, and / or 20004. The computing system may be or include a computing system 20271 as described herein with respect to Figure 9. The computing system may be, for example, a computer system 20063 as described herein with respect to Figure 4. The computing system may be, for example, a computer system 20064 as described herein with respect to Figure 4. The computing system may be or include surgical hub 20006, surgical hub system 20060 in Figure 4, computer-implemented bidirectional surgical system 20070 in Figure 5, surgical hub or computing device 20243 in Figure 7, surgical hub 20270 in Figure 9, console 20294 in Figure 10, and / or surgical hub 5104 in Figure 11, as described herein with respect to Figures 1 to 3. For example, the computing system may acquire surgical monitoring data associated with one or more surgical procedures. Surgical procedures may occur in one or more ORs.

[0180] Surgical monitoring data may be acquired via a surgical hub. For example, the surgical hub may acquire surgical monitoring data from various sensing systems, such as the wearable sensing system 20011 and / or the environmental sensing system 20015 described herein with reference to Figure 1. The surgical hub may acquire surgical monitoring data from the HCP monitoring device 35510, the environmental monitoring device 35512, the patient monitoring device 5124, and / or the modular device 5102, as described herein with reference to Figure 11. The surgical monitoring data may include situational awareness data described herein with reference to Figure 11.

[0181] The computing system may acquire surgical monitoring data from various sensing systems, such as the wearable sensing system 20011 and / or the environmental sensing system 20015 described herein with respect to Figure 1. Surgical monitoring data may also be acquired from the HCP monitoring device 35510, the environmental monitoring device 35512, the patient monitoring device 5124, and / or the modular device 5102, as described herein with respect to Figure 11.

[0182] A computing system can determine a virtual boundary associated with restricted access within an OR. The virtual boundary described herein may be, or may include, an area associated with restricted access within an OR. For example, an area associated with restricted access within an OR may be an enclosed area such as a circle, ellipse, rectangle, or square.

[0183] Figure 12 shows an example of one or more virtual boundaries associated with restricted access within an OR. As shown in Figure 12, OR36000 can contain one or more virtual boundaries, each of which can be associated with a corresponding restricted access.

[0184] A virtual boundary can be associated with an access restriction area for anesthesiologists within the OR. An access restriction area may be referred to herein as a virtual boundary area. For example, the virtual boundary area 36005 for anesthesiologists may be, or include, an access restriction area for anesthesiologists, as shown in Figure 12, for example. The virtual boundary area 36005 for anesthesiologists can restrict access to healthcare professionals (HCPs) within the OR and allow access to anesthesiologists, anesthesiology staff, and / or anesthesiology equipment. For example, an anesthesiologist may be permitted to enter, exit, and / or be positioned within the virtual boundary area 36005 for anesthesiologists, for example, during a surgical procedure. As described herein, a computing system can determine the virtual boundary associated with the access restriction area 36005 for anesthesiologists. The computing system can monitor the movements of HCPs within the OR. If the computing system determines that an HCP other than an anesthesiologist and / or anesthesiologist staff is in proximity to and / or attempts to enter or interact with the virtual boundary area 36005 for anesthesiologists, the computing system may send a notification to one or more HCPs. The notification may indicate that the HCP does not have access authorization to enter or interact with the virtual boundary area 36005 for anesthesiologists.

[0185] The virtual boundary area 36005 for the anesthesiologist may be a subset of one or more virtual boundary areas associated with the anesthesiologist, or may include such a subset. For example, the virtual boundary area 36010 may be associated with an area where the anesthesiologist is positioned during a surgical procedure. The virtual boundary area 36015 may be a virtual boundary area for anesthesiologist staff, such as anesthesiologist-nurses, and / or anesthesiologist equipment, such as a display for the anesthesiologist. The virtual boundary area 36020 may be a virtual boundary area where anesthesiologist instruments for the anesthesiologist may be positioned.

[0186] A virtual boundary area may be associated with an access restriction area for surgeons within the OR. For example, a virtual boundary area associated with an access restriction area 36025 for surgeons can be shown in Figure 12. The virtual boundary 36025 for surgeons may be an access restriction area for surgeons within the OR to enter, exit, and / or be in during a surgical procedure. The virtual boundary 36025 for surgeons may be, or include, an access restriction area where surgeons operate during a surgical procedure. The virtual boundary area 36025 for surgeons can restrict access to surgeons who are operating / will operate in the current surgical procedure. For example, a computing system can determine the virtual boundary area 36025 for surgeons within OR 36000. The computing system can monitor the movements of HCPs within the OR. For the current surgical procedure, the computing system can identify surgeons, such as the lead surgeon, who are about to perform the current surgical procedure. The computing system can enable the lead surgeon to access (e.g., enter, exit, or be positioned in) the virtual boundary area 36025 for surgeons. If the computing system determines that an HCP other than the identified surgeon is approaching or attempting to enter / interact with the virtual boundary area 36025 for surgeons, the computing system may send a notification to the HCP warning that the HCP does not have the necessary access authorization to enter or interact with the virtual boundary area 36025 for surgeons, and may restrict access.

[0187] As described herein, the virtual boundary area 36025 for surgeons may be, or include, one or more subsets of access restriction areas (not shown). For example, a subset of the virtual boundary area 36025 for surgeons may include a virtual boundary area for a surgeon performing the current surgical procedure of a surgery. A subset of the virtual boundary area 36025 for surgeons may include a virtual area for an assistant surgeon performing a surgical procedure. A subset of the virtual boundary area 36025 for surgeons may include a virtual boundary area for an observer, such as an intern. The computing system may enable identified HCPs, such as surgeons, assistant surgeons, and / or observers, to access their corresponding virtual boundary areas and restrict access to other HCPs.

[0188] A virtual boundary can be associated with an access restriction area for an operating table within the OR. For example, a virtual boundary associated with an access restriction area 36030 for an operating table may be shown in Figure 12. The virtual boundary area 36030 for an operating table may be an access restriction area that allows surgeons, such as a surgeon who is about to perform the current step of a surgical procedure, and / or a patient. For example, the virtual boundary area 36030 for an operating table may be a sterile area where a patient is positioned for surgical work. The virtual boundary area 36030 for an operating table may be sterilized. A computing system can restrict access to the virtual boundary area 36030 for an operating table. For example, a computing system may allow a patient to enter and be positioned in the virtual boundary area 36030 for an operating table. A computing system can control access to the virtual boundary area 36030 for an operating table and allow HCPs with access authorization to the virtual boundary area 36030 for an operating table to enter, exit, and / or be positioned therein. As described herein, the computing system may block access to an unauthorized HCP by sending a notification to enter or interact with the virtual boundary area 36030 for the operating table.

[0189] A virtual boundary may be associated with an access restriction area for a scrub nurse. For example, a virtual boundary associated with an access restriction area 36035 for a scrub nurse may be shown in Figure 12. As described herein, the virtual boundary area 36035 for a scrub nurse may be an access restriction area that allows a scrub nurse to enter, exit, and / or be positioned in, for example, during a surgical procedure. The computing system may allow a scrub nurse with access authorization to interact with or be within the virtual boundary area 36035 for a scrub nurse. The computing system may block an unauthorized HCP from accessing or interacting with the virtual boundary area 36035 for a scrub nurse and send a notification to the unauthorized HCP.

[0190] A virtual boundary may be associated with an access restriction area for operating room nurses. For example, a virtual boundary associated with access restriction area 36040 for operating room nurses may be shown in Figure 12. As described herein, the virtual boundary area 36040 for operating room nurses may be an access restriction area that allows, for example, operating room nurses to enter, exit, and / or be positioned within the virtual boundary area 36040 for operating room nurses. The computing system may enable operating room nurses with access authorization to interact with or be within the virtual boundary area for operating room nurses. The computing system may send a notification to an unauthorized HCP and block access by the unauthorized HCP to or interact with the virtual boundary area 36040 for operating room nurses.

[0191] A virtual boundary area may be associated with one or more access restriction areas for surgical instruments, sterile tray tables, and / or other surgical-related equipment. For example, a virtual boundary associated with an access restriction area for surgical instruments, sterile tray tables, and / or other surgical-related equipment 36045, 36050, 36055, 36060, 36065 can be shown in Figure 12. Embodiments of surgical-related equipment may include one or more of filtration systems, particle capture systems, and / or airborne contamination zones. As described herein, a computing system may enable an HCP with access authorization to enter, exit, interact with, and / or be positioned within a virtual boundary area for surgical instruments. For example, a scrub nurse with access authorization to enter the virtual boundary area 36045 of a sterile tray table may enter / exit the virtual boundary area 36045 of the sterile tray table during surgery. The computing system can block other HCPs who do not have access authorization to interact with (e.g., enter) the virtual boundary area 36045 of the sterile tray table from interacting with the virtual boundary area 36045 of the sterile tray table. The computing system can send a notification to the unauthorized HCP indicating that the HCP does not have access authorization to enter the virtual boundary area 36045 of the sterile tray table.

[0192] As shown in Figure 12, one or more virtual boundary areas can overlap. For example, the virtual boundary area 36025 for the surgeon may be inside the virtual boundary area 36030 for the operating table. In this example, during surgery, the scrub nurse can assist the surgeon by handing surgical instruments to the surgeon. The virtual boundary area 36035 for the scrub nurse can overlap with the virtual boundary area 36025 for the surgeon and / or the virtual boundary area 36030 for the operating table.

[0193] Virtual boundaries associated with surgical instruments and / or with the patient may change. For example, virtual boundaries associated with access restriction areas for surgical instruments may change based on whether the surgical instruments are in the off or on position (e.g., while the surgical instruments are being used during a surgical procedure). Virtual boundary areas associated with the patient may change during a surgical procedure, such as when the patient's body is moved, rotated, or repositioned.

[0194] Figure 13 shows another example of one or more virtual boundaries associated with restricted access within OR36100. As described herein, a virtual boundary may be associated with an access restriction area 36105 for an anesthesiologist. The virtual boundary area 36105 for anesthesiologists may include a virtual boundary area 36110 for anesthesiologist nurses. Because anesthesiologist nurses assist anesthesiologists during surgical procedures by handing over anesthesiologist equipment, anesthesiologist nurses may have access to enter, exit, and / or be positioned within the virtual boundary area 36105 for anesthesiologists.

[0195] As described herein, a virtual boundary may be associated with an access restriction area 36120 for the operating table and / or surgical area. The virtual boundary area 36120 for the operating table and / or surgical area may include a virtual boundary area 36115 for the surgeon. When a surgeon needs to operate on a patient who is on the operating table, the surgeon may be authorized to interact with the virtual boundary area 36120 for the operating table and / or surgical area (e.g., enter, exit, or position inside it).

[0196] As described herein, a virtual boundary may be associated with an access restriction area 36125 for nurses. Nurses may assist surgeons during surgical procedures. For example, nurses may hand surgical instruments to surgeons. Nurses may be authorized to interact with, enter, exit, and / or be positioned in, the virtual boundary area 36115 for surgeons, and / or the virtual boundary area 36120 for the operating table and / or surgical area.

[0197] A virtual boundary may be associated with an access restriction area 36130 for an assistant. The assistant may respond to one or more devices associated with HCP within the OR. For example, the assistant may respond to a cell phone and / or pager for the surgeon. Because the assistant's role may be non-surgical, the access authorization associated with the assistant may be at a minimum level (e.g., default level or basic level). For example, the assistant may not have access authorization to interact with, enter, exit, or be positioned in other virtual boundaries within the OR. In addition and / or alternatively, the virtual boundary area 36130 for the assistant may be used as a boundary or barrier for computing devices to monitor the assistant's movements. For example, if the assistant leaves and / or moves away from the virtual boundary area 36130 for the assistant, the computing system may send a notification to the assistant. The notification may indicate that the assistant may be outside the virtual boundary area 36130 designated for the assistant and may contaminate other virtual boundary areas that have sterile fields.

[0198] As described herein, a virtual boundary may be associated with an access restriction area 36135 for an observer. The observer may be positioned near the operating table (e.g., next to or adjacent to it) to observe the surgical procedure. The access authorization associated with the observer may be at a minimum level (e.g., default level or basic level). For example, the observer's role is to observe the surgical procedure and not interfere with it. The observer may not have access authorization to interact with, enter, exit, or position themselves in other virtual boundaries within the OR. In addition and / or alternatively, the virtual boundary 36135 for the observer may be used as a boundary or barrier for computing devices to monitor the observer's movements. For example, if the observer is in and / or leaves the virtual boundary area 36135 for the observer, the computing system may send a notification to the observer. The notification may indicate that the observer may be outside the virtual boundary area 36135 designated for the observer and may contaminate other virtual boundary areas with the sterile field.

[0199] As shown in Figure 13, the virtual boundary may be associated with an access restriction area 36140 for a surgical safety checklist (SSC). In the example, the computing system can determine whether the HCP is wearing personal protective equipment (PPE). If the computing system determines that the HCP is not wearing appropriate PPE for the surgical procedure, the computing system can send a notification to the HCP. The notification may indicate that the HCP is close to the SSC and the virtual boundary area 36140 for the HCP and may not be wearing PPE. In the example, the computing system can allow the HCP to enter the virtual boundary area 36140 for the SSC. The computing system can determine whether the HCP is wearing appropriate PPE for the surgical procedure (for example, when the HCP is inside the virtual boundary area 36140 for the SSC). If the computing system determines that the HCP is in proximity to the virtual boundary area 36140 of the SSC and is about to leave and / or move away from the virtual boundary area 36140 for the SSC, the computing system may send a notification to the HCP. The notification may indicate to the HCP that the HCP is not wearing appropriate PPE suitable for the surgical procedure and that the HCP may be at risk if he leaves and / or move away from the virtual boundary area 36140 for the SSC.

[0200] The virtual boundaries associated with access restriction areas described herein may be associated with virtual boundary areas (e.g., constant virtual boundary areas), selective boundary areas, and / or adaptive boundary areas. Figure 14 shows examples of virtual boundary areas 36200 (e.g., constant virtual boundary areas), selective virtual boundary areas, and / or adaptive virtual boundary areas in an OR. Virtual boundary areas (e.g., constant virtual boundary areas) may be associated with predefined and / or preconfigured equipment areas within an OR. For example, a virtual boundary area may be associated with a sterile field area 36205 for a surgical zone, a virtual boundary area 36210 for an anesthesiologist, a virtual boundary area 36215 for surgical instruments such as a high-energy equipment area, and so on. The virtual boundary areas described herein may be immutable (e.g., unadjusted or constant). Virtual boundary areas may be associated with high-risk areas. For example, a high-risk area may be associated with a surgical zone for infectious diseases, contagious diseases, and / or surgical procedures involving radioactive materials.

[0201] The virtual boundary area may be a predefined area surrounding an equipment area within an OR, based on, for example, surgeon preferences, hospital guidelines, or health organization protocols. The virtual boundary area may also be a predetermined area surrounding a high-risk area (for example, in accordance with the guidelines of a health organization such as the Centers for Disease Control and Prevention or the World Health Organization).

[0202] The virtual boundary area associated with the equipment area can be adjusted. The equipment area and / or high-risk area may be moved and / or shifted, for example, during a surgical procedure. Based on the moved and / or shifted area of ​​the equipment area and / or high-risk area, the virtual boundary area for the equipment may be moved and / or shifted accordingly (for example, together with the equipment).

[0203] Virtual boundary areas for equipment areas and / or high-risk areas can be adjusted, for example, automatically self-adjusting. In an example, surgical equipment can be moved and / or shifted when the surgical equipment is in the ON or OFF position (for example, as shown and described in Figure 15). The virtual boundary area associated with the surgical equipment can be adjusted accordingly. In an example, the patient's body can be repositioned and / or moved during surgery. Based on the repositioned and / or moved body of the patient, the virtual boundary area associated with high-risk areas can be adjusted accordingly.

[0204] Virtual boundary areas for equipment areas and / or high-risk areas can be adjusted by a computing system. For example, the computing system can determine whether a surgical instrument is in the ON position or the OFF position. When the computing system determines that a surgical instrument is in the ON position or the OFF position, it can adjust the virtual boundary area for the equipment area.

[0205] In this example, a computing system can monitor and / or track a patient's position during surgery. The computing system can determine if the patient is in a high-risk area. If the computer system determines that the patient's position has been rearranged, moved, adjusted, etc., the computer system can adjust a virtual boundary area for the high-risk area (e.g., associated with the patient's body).

[0206] Virtual boundary areas for equipment areas and / or high-risk areas can be adjusted by the HCP. In the example, surgical instruments may be moved by the HCP during a surgical procedure. The HCP can adjust the virtual boundary area for surgical instruments when the surgical instruments are being moved. In the example, the HCP may reposition the patient's body for the current surgical step or procedure. Based on the patient's repositioning, the HCP can adjust the virtual boundary areas associated with high-risk areas (e.g., associated with the patient's body) as appropriate.

[0207] The virtual boundaries associated with the access restriction areas described herein may have selective boundary areas. For example, an HCP, such as a surgeon, may define a virtual boundary area of ​​the surgical zone around a patient. As shown in Figure 14, such an area may be focused on the stomach or surrounding area of ​​the patient 36220. Surgical instruments, such as high-energy surgical instruments, may be activated inside the selective virtual boundary area, for example, on or around the patient's stomach 36220. Surgical instruments may be deactivated outside the selective virtual boundary area. For example, a computing system may identify a selective virtual boundary generated by the HCP. The computing system may monitor the location and / or position of the surgical instrument. The computing system may monitor the energy state of the surgical instrument. If the computing system determines that the surgical instrument is located and / or positioned within the boundary of the selective virtual boundary area, the computing system may generate a signal to activate the surgical instrument (e.g., transition from a low-energy state to a high-energy state). If the computing system determines that a surgical instrument is located and / or positioned outside the selective virtual boundary area, and / or is activated while the energy state of the surgical state is outside the selective virtual boundary area, the computing system may generate a signal to stop the surgical instrument. The computing system may send an alert to the HCP indicating that the surgical instrument was activated outside the selective virtual boundary area.

[0208] The virtual boundaries associated with the access restriction areas described herein may have adaptive boundaries. During a surgical procedure, the patient's body may be repositioned or moved. The selective virtual boundary of the surgical instrument (e.g., generated by the HCP) may be modified or adjusted when repositioning and / or movement is detected. For example, as illustrated in Figure 14, the patient's body may be repositioned (e.g., repositioned to its side), the selective virtual boundary area may be reduced, and / or the patient's arm may be within the selective virtual boundary area which allows the surgical instrument to be in a high-energy state. The adaptive virtual boundary area 36225 may be generated based on the change and / or repositioning of the patient's body, and the surgical instrument may not be activated in a high-energy state relative to the patient's repositioned arm area.

[0209] The computing system can generate adaptive virtual boundary areas and / or identify adaptive virtual boundary areas generated by the HCP. For example, the computing system can monitor the patient's position and selective virtual boundary areas. If the computing system detects and / or determines that the patient's position has changed and / or that the selective virtual boundary area needs to be adjusted, the computing system can generate an adaptive virtual boundary area to adapt to the change. The adaptive virtual boundary area may be temporary and may be turned off when the patient's body is changed (e.g., returned to its previous position). The HCP can generate adaptive virtual boundary areas. When the HCP repositions the patient's body, the HCP can indicate the adaptive virtual boundary area to the computing system, and the computing system can identify the adaptive virtual boundary area defined by the HCP.

[0210] The virtual boundaries described herein may be pre-configured by the Health Care Planner (HCP). For example, an HCP preparing for a surgical procedure may pre-configure one or more virtual boundary areas within the operating room (OR). These virtual boundary areas may be based on the arrangement of surgical instruments, the arrangement of the operating table, the arrangement of sterile surgical tray tables, an anesthesiologist's zone, etc. The HCP may manually input and / or adjust the virtual boundary areas.

[0211] The computing system can determine one or more virtual boundary areas based on historical data associated with HCPs and / or surgical procedures. For example, the computing system can determine that the same surgical procedure was performed by the same HCP. Based on historical data, the computing system can pre-configure one or more virtual boundary areas. Historical data may include, or be based on, data from different ORs, HCP preferences, preoperative data, etc.

[0212] As described herein, surgical procedures may be related to infections, airborne contamination, etc. Virtual boundaries may be created for purposes such as infection control and tracking, and airborne contaminant zone monitoring. For example, virtual boundaries may be associated with filtration system areas, particle capture system areas, airborne contamination zones, etc.

[0213] The computing system can identify HCPs within the OR. As described herein, the computing system can identify HCPs based on ID tags, camera feeds, facial recognition, etc. Based on the identification of HCPs within the OR, the computing system can determine access authorizations associated with the HCPs. The computing system can retrieve, download, and / or determine access authorizations associated with the HCPs. Access authorizations may be, or include, access authorization information for each HCP to interact with, enter, exit, and / or be located within the virtual boundary area.

[0214] The computing system can monitor the movement of identified HCPs within the OR. For example, the computing system can track the movement of HCPs. The computing system can monitor the movement of HCPs based on, for example, a camera system within the OR, RFID tags associated with the HCPs, etc.

[0215] Based on the monitored movement of the HCP and the access authorization associated with the HCP, the computing system can determine whether the HCP has access authorization to interact with the virtual boundary associated with the restricted access area. As described herein, the virtual boundary associated with the restricted access area may be associated with surgical equipment areas, high-risk areas, infection control areas, airborne contamination areas, and the like.

[0216] If the computing system determines that an HCP is interacting with a virtual boundary associated with an access restriction area, or does not have access authorization to enter that virtual boundary (e.g., is not authorized to enter), and is in proximity to the virtual boundary area, the computing system may send a notification to the HCP. The notification to the HCP may indicate that the HCP is interacting with the virtual boundary area or is not authorized to enter the virtual boundary area. The computing system may send notifications to devices associated with the HCP, which may be one or more of the following: visual notifications, audible notifications, haptic notifications, or augmented reality notifications, or may include them. The computing system may send notifications to displays in the OR, and / or to one or more other devices associated with other HCPs in the OR.

[0217] If, after interacting with a virtual boundary area or sending a notification to an HCP that is not authorized to enter a virtual boundary area, the computing system determines that the HCP continues to approach the virtual boundary area and / or continues to move closer to the virtual boundary area, the computing system may send notifications to displays within the OR and / or devices associated with one or more other HCPs within the OR, thereby alerting one or more other HCPs within the OR.

[0218] If the computing system determines that the HCP has access authorization to enter the virtual boundary associated with the access restriction area, the computing system may allow the HCP to interact with the virtual boundary area, enter the virtual boundary area, leave the virtual boundary area, and / or be positioned within the virtual boundary area. For example, the computing system may skip sending a notification to the HCP.

[0219] As described herein, the virtual boundary may be associated with a static (e.g., constant) boundary around the sterile field area. The computing system can monitor the movements of HCPs within the OR and send notifications to one or more HCPs who are not wearing appropriate sterile equipment. For example, the computing system may send a notification to an HCP who is not wearing sterile surgical gowns and / or is not authorized to enter the virtual boundary of the sterile field area when the HCP is approaching the vicinity of the virtual boundary area for the sterile field. After sending a notification to an HCP, if the computing system determines that the HCP is attempting to interact with (e.g., enter) the virtual boundary area of ​​the sterile field area, the computing system may send a notification to displays in the OR and / or in other HCPs, such as scrub nurses, and / or broadcast the notification throughout the OR to block or intervene to prevent the unauthorized HCP from interacting with or entering the virtual boundary area of ​​the sterile field area. By sending notifications to block unauthorized HCPs from interacting with the virtual boundary area, the computing system can prevent potential contamination in the virtual boundary area of ​​the sterile field area.

[0220] As described herein, a virtual boundary may be associated with an area for treating infectious patients. A dynamic, constant virtual boundary may be created around an area from which aerosolized particles may be released. A virtual boundary for an area may be associated with a filtration system area and / or a particle capture system area. A computing system may monitor the movements of HCPs within the OR. The computing system may identify HCPs with PPE and / or certificates or access authorizations within the OR to handle infectious and / or infectious surgical procedures, such as the treatment of infectious patients. If the computing system determines that an HCP does not have PPE, certificates, and / or access authorizations to enter the virtual boundary area for infectious patients, the computing system may send a notification to the HCP. The computing system may use the notification to warn the HCP to avoid the virtual boundary area for infectious patients, for example, via a notification.

[0221] The virtual boundary associated with an access restriction area may be an equipment area, or may include an equipment area. As described herein, the virtual boundary of an equipment area may be a predefined boundary surrounding the equipment area. When equipment moves around during a surgical procedure, the virtual boundary around the equipment may remain constant. For example, a certain predetermined virtual boundary area may be generated for monitors, operating tables, sterilization tables, surgical instruments, etc. A certain predetermined virtual boundary may exist during a surgical procedure. A computing system may monitor (e.g., track) the movement of equipment during a surgical procedure. The computing system may exclude unauthorized HCPs from the virtual boundary area for equipment when equipment moves around during a surgical procedure. The computing system may enable authorized HCPs to interact with, enter, exit, and occupy a certain predefined virtual boundary area for equipment.

[0222] Surgical instruments, such as the robotic arm illustrated in Figure 15, can send notifications to one or more HCPs within the OR. For example, before and / or during a movement, the surgical instrument can indicate to the HCP in the OR that a movement is occurring. The surgical instrument and / or computing system can indicate a virtual boundary area where the movement ends and display the adjusted area to the HCP. For example, the computing system may provide a visual indicator (e.g., a colored laser) to indicate the adjusted position of the virtual boundary area associated with the surgical instrument. The visual indicator can allow unauthorized HCPs to be excluded from the virtual boundary area of ​​the surgical instrument and, if necessary, allow authorized HCPs to be repositioned within the virtual boundary.

[0223] The HCP and / or equipment may block the movement of a surgical instrument, for example, the movement of its end. If a surgical instrument determines that it cannot reach a complete end movement, it may send a notification to the HCP. For example, the surgical instrument may provide an audible alert to the HCP to move out of the way, ask the HCP to move any equipment blocking the surgical instrument from reaching its final movement, and / or ask the HCP to readjust the surgical instrument. The surgical instrument may provide a visual indication to the HCP, for example, to ensure that the surgical instrument does not come into contact with the HCP and / or equipment, does not interfere with the current surgical procedure, and / or causes damage.

[0224] As described herein, virtual boundaries can be associated with selective virtual boundary areas. For example, selective virtual boundary areas can be based on the use of equipment within an OR. Selective virtual boundary areas can be adjusted based on the energized state of surgical instruments. For example, if a computing system determines that a surgical instrument is in off mode and / or low energized, the computing system can determine or identify a selective virtual boundary area that allows access to one or more HCPs, such as a surgeon, scrub nurse, or observer. If a computing system determines that a surgical instrument is in a high energized state (for example, if a surgeon is using the surgical instrument during the current surgical procedure), the computing system can adjust the selective virtual boundary area to exclude other HCPs who are not using the surgical instrument. For example, if a surgical instrument is in a high energized state, the computing system can shut out other HCPs and allow access to the surgeon performing the current surgical procedure.

[0225] A selective virtual boundary area associated with a surgical instrument can be set based on which arm is active and / or where the arm of the surgical instrument is positioned and can reach. For example, if the surgical instrument (e.g., the arm of the surgical instrument) is in off mode and / or not moving, the selective virtual boundary area may be excluded (e.g., temporarily and / or until the arm is operational).

[0226] Figure 15 shows an exemplary surgical instrument having an arm in OR36300. The surgical instrument may be in off mode, and a virtual boundary area associated with off mode 36305 may be generated. The virtual boundary area of ​​the arm of the surgical instrument may be in the off position 36315. When the surgical instrument is operating and / or in on mode, a selective virtual boundary area 36310 around the surgical instrument associated with on mode may be generated. The selective virtual boundary area 36310 around the surgical instrument for on mode has a larger area than the virtual boundary area 36305 associated with off mode, allowing the surgical instrument to move around. A selective virtual boundary area 36320 of the arm of the surgical boundary associated with the on position may be generated. The selective virtual boundary 36320 can be adjusted based on the space that the arm can occupy while in use (e.g., while operating). For example, the selective virtual boundary area 36320 of the arm of the surgical instrument in on mode can cover the sweeping motion of the arm's movement.

[0227] For example, the surgical instrument shown in Figure 15 may be an imaging system. For example, a hybrid OR may be constructed around the imaging device. For example, the imaging device may be a non-mobile imaging device such as a computerized radiography (CT) device or a magnetic resonance (MR) device. The imaging device may include high-energy imaging components that need to be avoided by a high-risk compensator (HCP) via a virtual boundary area. The imaging device may include a boom and / or other movable components that may require manual or powered repositioning. The support column of the imaging device may have a selective virtual boundary area when the imaging device is in use. When the support column is in use, the selective virtual boundary area may be generated to protect the HCP from inadvertent interaction with the imaging device and / or any hazards associated with the imaging device.

[0228] The imaging device may be, for example, a mobile imaging system, as shown in Figure 15. The mobile imaging system can be moved in and / or inside the OR. The mobile imaging system may be operated to the area where imaging is required, for example, via a stabilizing arm. The positioning range of the mobile imaging system and / or the exposure area may have corresponding selective virtual boundary areas.

[0229] Exemplary mobile imaging systems may include fluoroscopy such as C-arm radiography, 3D imaging, and / or ultrasound imaging. As shown in Figure 15, the stabilizing arm of the mobile imaging device may have a selective virtual boundary area when the imaging device is in use. The selective virtual boundary 36325 for the stabilizing arm of the mobile imaging device can be focused on the arm or around the arm of a patient requiring re-imaging review. When the stabilizing arm of the imaging device is in motion, the selective virtual boundary area 36325 can be generated to protect HCP from inadvertent interaction with the imaging device and / or any hazards associated with the imaging device.

[0230] The virtual selection boundary associated with the imaging devices described herein can be contoured on the floor of the OR. The virtual selection boundary associated with the imaging device may include a vertical boundary or a three-dimensional boundary. For example, the mobile system shown in Figure 15 may require a spherical boundary of, for example, 2 feet away from the floor, which can be used to focus on an imaging site on the patient's arm for review during orthopedic surgery.

[0231] The virtual boundary associated with an access restriction area can be adjusted based on one or more conditions. For example, as described herein, the virtual boundary for radioactive devices and / or high-energy imaging equipment can be adjusted, for example, based on power level and / or imaging focus, to minimize inadvertent exposure to HCPs within the OR. The virtual boundary area may be based on the physical boundary of the equipment, the size of the OR, and / or the energized state of the equipment (e.g., power level).

[0232] The virtual boundary associated with the access restriction area can be adjusted based on the surgical procedure and / or the possibility of infection. For example, when intubating or extubating, a patient may have a virtual boundary area generated around the patient's head and / or shoulder area. The virtual boundary area may be small and / or encompassed by a filtration system during surgical procedures such as bronchoscopy. The virtual boundary area can be adjusted during intubation or extubation. For example, during intubation or extubation, the virtual boundary area may be adjusted to be larger, for example, due to the possibility of aerosols from respiration contaminating a larger zone. The virtual boundary around the patient's head and / or shoulder area may include an adaptive virtual boundary area. For example, the adaptive virtual boundary around the patient area may be adjusted dynamically, for example, when the patient coughs or forcibly expels air (e.g., in response), and the aerosol and virtual boundary area may be adjusted by expanding the restriction area.

[0233] The computing system can notify HCPs within the OR if the adaptive virtual boundary area for surgical instruments changes. For example, the computing system can notify HCPs that the virtual boundary area for surgical instruments, e.g., the adaptive virtual boundary area, has changed. The computing system can monitor authorized HCPs within the virtual boundary area for exposure levels and / or verify that the PPE being used by HCPs is appropriate.

[0234] Virtual boundary areas may be drawn using laser light. HCPs within the OR can visually recognize the location of virtual boundary areas within the OR by observing the laser light. Virtual boundary areas may have different colors to represent different types of virtual boundary areas. For example, certain virtual boundary areas may have different laser colors than selective and / or adaptive virtual boundary areas. Different laser light can provide information about the hazard level, power level, etc., associated with the virtual boundary area.

[0235] The virtual boundary area may be defined by the layout of the OR, the location / position of surgical instruments, and / or the equipment area within the OR. As described herein, the virtual boundary area may be adjusted based on the movement of surgical instruments and / or equipment during a surgical procedure. The virtual boundary area may also be adjusted based on the repositioning of the patient (e.g., moving the patient laterally) or the repositioning of the OR table and sterile surgical tray table.

[0236] The computing system may send a notification to an unauthorized HCP when it determines that the unauthorized HCP is approaching, near, and / or close to a virtual boundary associated with an access-restricted area within the OR. The computing system may also send notifications to authorized HCPs regarding boundary interactions, for example, to ensure the security of the HCP. For example, the computing system may send notifications to authorized HCPs before and / or after entering or leaving a virtual boundary. Notifications to HCPs (e.g., authorized and / or unauthorized HCPs) can prevent accidental or inadvertent interactions with virtual boundary areas.

[0237] As described herein, a computing system may send notifications to an HCP within the virtual boundary of an access-restricted area, or when approaching the virtual boundary area. The computing system may use one or more of the following: haptic notifications, visual notifications, audible notifications, and visual-tactile notifications. Notifications may be modalities and / or feedback, or may include them. Notifications may provide warnings, provide awareness of the surroundings, such as the location of the virtual boundary of the access-restricted area, prevent collisions with other things or equipment, such as sterile areas, and / or provide instructions if an individual is near or in violation of a controlled area or boundary.

[0238] The computing system can provide visual feedback. The computing system can provide visual indicators and display virtual boundaries of restricted access areas or zones within the OR. The computing system can use a laser projector to project (e.g., outline) virtual boundaries of restricted areas or zones onto the floor, walls, and / or within the OR. For example, the computing system can use a laser projector to provide visual indications of virtual areas and / or zones to HCPs within the OR. The computing system can use different colors to represent different virtual boundaries associated with restricted areas / zones. For example, the computing system may use blue to indicate virtual boundaries for sterile field areas, orange to indicate virtual boundaries for high-energy equipment, and purple to indicate virtual boundaries for anesthesiologist equipment. The computing system can change the color of light when an unauthorized HCP is approaching the virtual boundary of an restricted access area. For example, the computing system may change the light to yellow when an unauthorized HCP is near the virtual boundary of an restricted access area, and to red when an unauthorized HCP enters / interacts with the virtual boundary of an restricted access area.

[0239] The computing system may use one or more laser markings for virtual boundaries associated with access-restricted areas / zones based on one or more of the roles, licenses, or certificates associated with HCPs within the OR. For example, a virtual boundary for a sterile area / zone may show green when an HCP with correct verification and / or valid credentials is near or inside the virtual boundary. If an unauthorized HCP with incorrect verification and / or inappropriate credentials is near or inside the virtual boundary, the computing system may use red to provide a warning and / or alert to the unauthorized HCP.

[0240] A computing system can generate a virtual boundary for an access-restricted area, and the virtual boundary can be made visible using an AR device such as augmented reality (AR) glasses. For example, the computing system may generate a colored zone as described herein, and / or generate laser light on the floor, and the HCP can see the virtual boundary using an AR device.

[0241] The computing system can display the OR. For example, the computing system can display one or more of the following: the OR, HCPs within the OR, the movements of HCPs within the OR, surgical equipment, etc. The computing system can display one or more virtual boundaries of access-restricted areas. The computing system can highlight virtual boundary areas if an unauthorized HCP is approaching and / or interacting with a virtual boundary area. The computing system can provide an overview of the OR to HCPs within the OR. The OR display and the displayed information can enable HCPs, such as surgeons, to reinforce one or more virtual boundary areas and / or disable highlighted virtual boundary areas.

[0242] The computing system can provide audible feedback. The computing system can provide audible sounds or information to HCPs who are near or interacting with the virtual boundary of an access-restricted area. For example, the computing system can broadcast audible information about a virtual boundary area when an authorized HCP is approaching the boundary area, or when an unauthorized HCP is approaching or interacting with the virtual boundary area. The computing system can provide different sounds based on defined virtual boundary areas. For example, the computing system may have different sounds for virtual boundaries for sterile field areas, patient areas, equipment areas, and critical areas.

[0243] The computing system can provide different sound levels based on different virtual boundary areas and / or the proximity of HCPs to those virtual boundary areas. For example, if an unauthorized HCP is approaching a virtual boundary area, the computing system can provide the HCP with short, quiet audible feedback. As the unauthorized HCP approaches the virtual boundary area, the audible feedback can become longer and louder (e.g., increasing the level of audible feedback). The computing system can provide audible feedback to the corresponding HCP, for example, through a device associated with the HCP. If the computing system is broadcasting audible feedback within the OR, the computing system can, for example, noise-canceling the audible feedback to other HCPs to minimize disruption.

[0244] The computing system can provide haptic feedback. The HCP may be wearing one or more devices such as a phone, smartwatch, or smart tracker. For example, the HCP may be wearing devices on their wrists and / or ankles (e.g., gloves, hands, shoes, feet, etc.). The HCP may be wearing devices on each limb that may have wristbands and / or anklebands. The computing system can provide haptic feedback to limbs that are at risk of contacting the virtual boundary for an access restriction area. For example, if the left foot of an unauthorized HCP is near the virtual boundary of a sterile field area, the computing system can send haptic feedback to a device on the left ankle of the unauthorized HCP. The computing system can send haptic feedback to the left ankle device and provide notification that the HCP cannot step or walk in that direction (e.g., to the left).

[0245] The computing system can provide different levels of haptic feedback. The computing system can increase or decrease the level of haptic feedback when an HCP is approaching and / or moving away from the virtual boundary of an access-restricted area. For example, if an unauthorized HCP is approaching the virtual boundary of an access-restricted area, the computing system can increase the intensity of the haptic feedback and / or increase the pulse of the haptic feedback. If an unauthorized HCP is moving away from the virtual boundary of an access-restricted area, the computing system can decrease the intensity of the haptic feedback and / or decrease the pulse of the haptic feedback.

[0246] If the computing system determines that an unauthorized HCP is ignoring haptic feedback to a specific device (e.g., a left ankle device), the computing system may send haptic feedback to one or more devices worn by the unauthorized HCP.

[0247] The computing system may combine one or more of the feedbacks described herein to provide a visual-haptic mixed reality (VHMR).

[0248] A computing system can monitor HCPs and determine access authorization for HCPs. For example, a computing system can determine the permissibility of crossing the virtual boundary of an access-restricted area. A computing system can detect environmental parameters and / or personal parameters to determine, for example, the permissibility of crossing a virtual boundary area. For example, a computing system can detect environmental parameters and / or personal parameters based on one or more of the following: credentials, certificates, and / or other access control level information associated with the HCP.

[0249] The computing system can perform sterility monitoring of HCPs. For example, if the computing system determines that an HCP has passed the sterility test, the computing system can allow the HCP to enter the virtual boundary of the access-restricted area. If the computing system determines that an HCP has failed the sterility test, the computing system can revoke the access authorization associated with the HCP and deny the HCP access to the virtual boundary of the access-restricted area.

[0250] Because surgical procedures carry a risk of infection, HCPs in operating rooms (ORs) can receive extensive training in techniques for preventing surgical infections. For example, HCPs in an OR (e.g., surgeons or scrub nurses) can be mindful of how to properly put on and take off sterile protective equipment such as surgical gowns and / or gloves, how to wash their hands thoroughly to reduce the risk of bacterial contamination, and / or how to manage surgical instruments / tools to prevent contamination of sterile instruments or bandages. Operating room nurses who are not wearing fully sterile protective equipment (e.g., scrub nurses and / or surgeons wearing less sterile protective equipment) may not be authorized to enter virtual boundaries of restricted access areas, such as sterile areas, within the OR. Computing systems can monitor the movements of operating room nurses within the OR and prevent them from entering or interacting with the virtual boundaries of sterile field areas, thereby preventing contamination of the sterile area around the operating table.

[0251] The computing system can monitor the HCP. For example, the computing system can monitor the HCP pre- and post-operatively to verify that necessary steps have been completed. If the computing system determines that the HCP failed to complete steps such as proper cleaning before entering the OR or after performing surgery, the computing system can send a notification to the HCP and / or revoke the HCP's access authorization. The computing system can temporarily revoke the HCP's access authorization, for example, until the steps are completed.

[0252] The computing system can monitor equipment during surgical procedures. For example, the computing system can monitor surgical gloves for any signs of damage or surgical gowns for tears. If the computing system detects a tear or damage, it can send a notification to the HCP (Health Care Planner) to replace the equipment.

[0253] A computing system can monitor battery movement to surgical devices. For example, battery packs for surgical devices (e.g., lithium and / or lithium-ion battery packs) may not be sterile. The battery pack may have an embedded RFID chip that tracks sterile movement. For example, a battery pack may enter an OR for surgical procedures. The computing system can receive alerts from the battery pack (e.g., via an RFID beacon) and can identify the battery pack entering the OR. The computing system can track the battery pack and send notifications to the HCP based on the virtual boundaries of the access-restricted area within the OR.

[0254] The computing system can monitor the remediation steps associated with the battery bag. If a remediation step is missed or performed incorrectly, the computing system can alert the HCP to the potential problem.

[0255] The computing system can identify virtual boundaries of access-restricted areas for surgical instruments. Surgical instruments may be capable of performing surgical procedures using high-energy states. The computing system can detect the energy state associated with surgical instruments during surgical procedures. The computing system can identify surgical procedures of a surgery. For example, the computing system can identify surgical procedures based on surgical monitoring data. A surgical procedure may be the current surgical procedure, the next surgical procedure, or include both.

[0256] Based on the identified surgical procedure and the detected energized state, the computing system can adjust the virtual boundary area of ​​the surgical instrument. For example, if the computing system detects that a surgical instrument is in a highly energized state and matches a surgical procedure, the computing system can adjust the virtual boundary of the surgical instrument, for example, by increasing the virtual boundary area. If the computing system detects that the energized state exceeds a threshold level, the computing system can increase the virtual boundary area for the surgical instrument. If the computing system detects that the energized state is below a threshold level, the computing system can decrease the virtual boundary area for the surgical instrument.

[0257] If the computing system determines that a surgical instrument is in a highly energized state but does not correspond to the surgical procedure, for example, if an anomaly exists, the computing system can adjust the virtual boundary area of ​​the surgical instrument. For example, if the computing system detects that a surgical instrument is in a highly energized state when the current step of the surgical procedure does not require the surgical instrument to be turned on and / or used, the computing system can increase the virtual boundary area of ​​the surgical instrument. The computing system can send a notification to the HCP, for example, to prevent accidental contact with the surgical instrument.

[0258] Figure 16 shows an exemplary notification from the computing system regarding an unauthorized energy level change of a surgical instrument. As shown in Figure 16, the computing system can identify one or more virtual boundaries of the access restriction area 36400 within the OR. One or more virtual boundaries of the access restriction area within the OR may include virtual boundary area 36420 for the anesthesiologist area, virtual boundary area 36410 for the operating table area, virtual boundary area 36415 for the surgical instrument area, and / or virtual boundary area 36425 for the patient's surgical area. As described herein, the computing system can monitor the movements of the HCP within the OR. The computing system can determine whether the HCP has access authorization to enter, exit, interact with, or be within the virtual boundaries of the access restriction area within the OR. The computing system can determine the HCP's access authorization based on one or more of the following: the HCP's role during the surgical procedure, the assignment associated with the HCP during the surgical procedure, the PPE associated with the HCP, the biomarker measurements associated with the HCP, or the restricted area type.

[0259] The computing system can monitor the energy levels of surgical instruments within the OR. If the computing system detects a change in the energy level of a surgical instrument, it can determine that the change was made by an authorized HCP. If the computing system determines that the change was made by an unauthorized HCP who has entered the virtual boundary area 36415 for the surgical instrument, the computing system can send a notification. The computing system can send the notification to display 36405 to inform other HCPs that an unauthorized HCP has inadvertently altered the energy level of a surgical instrument. The computing system can send a notification to display 36405 and / or to the unauthorized HCP indicating that the HCP is near the virtual boundary 36415 for the surgical instrument or does not have authorization to reside within the virtual boundary.

[0260] If another HCP, such as a surgeon, sees a notification regarding an unauthorized HCP and / or an unauthorized energy level change caused by an unauthorized HCP, the surgeon, for example, the lead surgeon, can adjust the access authorization associated with the unauthorized HCP. For example, the unauthorized HCP may be another surgeon assisting the lead surgeon. The assistant surgeon may have access authorization to be within the virtual boundary 36410 for the operating table area. The assistant surgeon may not have access authorization to enter or be within the virtual boundary area 36415 for surgical instruments. The lead surgeon can adjust the assistant surgeon's access authorization. For example, after seeing a notification on display 36405, the lead surgeon can adjust the assistant surgeon's access authorization to allow the assistant surgeon to enter or be within the virtual boundary area 36415 for surgical instruments.

[0261] Another HCP may adjust the access authorization of an unauthorized HCP based on the emergency. For example, during an emergency where the leading surgeon requires an additional hand to operate a surgical instrument, the leading surgeon may adjust the access authorization of an assistant surgeon so that they enter or are within the virtual boundary area 36415 for the surgical instrument, as described herein.

[0262] The computing system can identify equipment, such as surgical instruments, associated with surgical procedures within the OR. For example, the computing system can download a surgical plan created by the HCP for pre-operative preparation and identify one or more surgical instruments to be used for the surgical procedure. The computing system can detect control inputs from the HCP for controlling the surgical instruments. These control inputs may involve turning the surgical instruments on or off, adjusting the energy level of the surgical instruments, or adjusting the position of the surgical instruments.

[0263] A computing system can determine the access control level of an HCP associated with a surgical instrument. For example, the access control level associated with an HCP may include whether the HCP has authorization to control the surgical instrument. The access control levels associated with an HCP can be hierarchical. For example, a hierarchical access control level (e.g., a first hierarchical access control level) may be turning the surgical instrument on or off. Another hierarchical access control level (e.g., a second hierarchical access control level) may be adjusting the energy level of the surgical instrument. For example, a second hierarchical access control level associated with adjusting the energy level of a surgical instrument may be a higher hierarchical level than a first hierarchical access control level associated with turning the surgical instrument on or off.

[0264] The computing system can determine whether to enable the control input detected by the HCP. The computing system can determine whether to enable the control input by the HCP based on the access control level associated with the HCP. If the computing system determines, based on the access control level, that the HCP is authorized to control the surgical instrument (for example, that the HCP is authorized to enable the control input for controlling the surgical instrument), the computing system can enable the control input by the HCP and control the surgical instrument. If the computing system determines, based on the access control level associated with the HCP, that the HCP is not authorized to control the surgical instrument (for example, that the HCP is not authorized to enable the control input for controlling the surgical instrument), the computing system can block the control input by the HCP for controlling the surgical instrument.

[0265] The computing system may send an alert to a device associated with a surgical instrument if the computing system blocks a control input from the HCP attempting to control the surgical instrument. The alert may be, or may include, a notification to the HCP informing it that the control input from the HCP has been blocked, and / or access control level information associated with the HCP. The computing system may also send an alert to a display in the OR. For example, as shown in Figure 16, the computing system may send an alert to display 36405 in the OR. The alert may indicate that the control input from the HCP has been blocked.

[0266] The computing system may be or include an HCP monitoring system such as HCP monitoring systems 20000, 20002, 20003, or 20004 as described herein with respect to Figures 1 to 3. The computing system may be a computing system operably connected to HCP monitoring systems 20000, 20002, 20003, and / or 20004. The computing system may be or include a computing system 20271 as described herein with respect to Figure 9. The computing system may be or include a computer system 20063 as described herein with respect to Figure 4, for example. The computing system may be or include a computer system 20064 as described herein with respect to Figure 4, for example. The computing system may be, or include, the surgical hub 20006, the surgical hub system 20060 in Figure 4, the computer-implemented bidirectional surgical system 20070 in Figure 5, the surgical hub or computing device 20243 in Figure 7, the surgical hub 20270 in Figure 9, the console 20294 in Figure 10, and / or the surgical hub 5104 in Figure 11, as described herein with respect to Figures 1 to 3. For example, the computing system may acquire surgical monitoring data associated with one or more surgical procedures. Surgical procedures may occur in one or more ORs.

[0267] As described herein, a computing system can monitor the movements of an HCP. The computing system can determine that an HCP is in close proximity to the operating table. The computing system can determine, for example, that an HCP is not authorized to control surgical instruments based on the access control level associated with the HCP. The computing system can send an access control level adjustment message to another HCP, such as a surgeon. The access control level adjustment message can query whether the access control level associated with the HCP requires adjustment based on the HCP's proximity to the operating table. The computing system can receive an access control level adjustment request from another HCP, such as a surgeon. The access control level adjustment request can indicate a change to the access control level associated with the HCP. Based on having received an access control level adjustment request from another HCP, the computing system can adjust the HCP's access control level. As described herein, the access control level adjustment request can enable the HCP to control surgical instruments (e.g., turn surgical instruments on / off and / or adjust the energy level associated with surgical instruments). The computing system can send an access control level adjustment notification to the HCP. The computing system can send the access control level adjustment notification to a display in the OR. Access control level adjustment notifications can indicate that an HCP has authorized a change in the access control level (for example, by a surgeon).

[0268] The computing system can adjust the control access level for surgical instruments associated with the HCP based on the surgical process in a surgical procedure. The computing system can identify the current surgical process. Based on the current surgical process, the computing system can determine whether to adjust the access control level associated with the HCP. For example, in the initial stages of a surgical procedure, the control access level for surgical instruments associated with the surgeon may be to turn the surgical instruments on or off. If the surgeon attempts to adjust the energy level associated with the surgical instruments in the initial stages of a surgical procedure, the computing system can block the control input from the surgeon.

[0269] The computing system can adjust the access control level associated with the HCP so that the HCP can control the surgical instruments associated with the current surgical procedure in the surgical process. For example, the computing system can determine that the current surgical procedure involves making an incision using a surgical instrument. The computing system can adjust the surgeon's control level. The surgeon can adjust the energy level of the surgical instrument, for example, by adjusting the surgical instrument from a low-energy state to a high-energy state, thereby enabling the incision to be made.

[0270] After the HCP completes the current surgical procedure, the computing system can readjust the access control level associated with the HCP to match the next surgical procedure. For example, after the surgeon makes an incision for the current surgical procedure and the computing system determines that the next surgical procedure does not involve a high-energy stage for surgical instruments, the computing system can readjust the access control level associated with the HCP to cut off the surgeon's control input and, for example, increase or maintain the high-energy level for surgical instruments.

[0271] The computing system can adjust the access control level of the HCP associated with an instrument based on the instrument's energized state. The computing system can identify the energized state of surgical instruments. If the computing system identifies that a surgical instrument is in a highly energized state, it can adjust the HCP's access control level and block the HCP's control input to control the surgical instrument. For example, to prevent accidental control of a surgical instrument, the computing system can block the control input from the scrub nurse when it determines that the surgical instrument is in a highly energized state.

[0272] If the computing system identifies that a surgical instrument is in a low-energy state or is turned off, the computing system can adjust the access control level of the HCP and enable control inputs from the HCP to control the surgical instrument. For example, when the computing system determines that a surgical instrument is in a low-energy state or is turned off, it can allow the scrub nurse to turn the surgical instrument on or off.

[0273] The computing system can adjust the control access level of the HCP based on biomarker measurements associated with the HCP. Biomarkers associated with the HCP may be one or more of the following: fatigue level, stress level, or the amount of time the HCP has been operating, or may include all of these. The computing system can monitor biomarkers associated with the HCP, such as those of a surgeon. Based on the monitored biomarkers of the HCP, the computing system can determine whether to adjust the access control level associated with the HCP. If biomarkers associated with the HCP indicate that the HCP has increased fatigue levels, increased stress levels, or exceeded a surgical time threshold, the computing system can block control input from the HCP to control surgical instruments. The computing system can send alerts to the HCP indicating increased fatigue levels, increased stress levels, or exceeded surgical time thresholds.

[0274] If biomarkers associated with the HCP indicate that the HCP has a reduced fatigue level, reduced stress level, or is below a threshold for surgical time, the computing system can enable control input from the HCP to control surgical instruments. The computing system can send alerts to the HCP indicating the current fatigue level, current stress level, or current surgical time associated with the HCP.

[0275] In the example, a computing system can receive measurement data associated with HCP from a sensing system. Based on the measurement data, the computing system can determine the elevated fatigue level associated with HCP. The measurement data may indicate that HCP, such as a surgeon's, may be overestimating changes in input, sometimes called overcorrection, for perceived errors. The computing system can interpret repeated corrections, overcorrections, or oscillation responses as indicators of fatigue and / or elevated fatigue levels associated with HCP. For example, based on biomarker measurements of HCP, the computing system can determine values ​​associated with HCP hydration / dehydration. Dehydration can affect energy levels and cause fatigue. Low body fluid levels tend to increase heart rate. The computing system can analyze heart rate data in relation to hydration levels and distinguish stress and other cardiac elevation events from hydration. The computing system can employ baseline measurements to distinguish acute events from ongoing chronic events and distinguish fatigue from dehydration.

[0276] In the example, the computing system can receive measurement data from a sensing system associated with the HCP within the OR (e.g., a sensing system associated with the surgeon). The measurement data may indicate that the HCP has an increased stress level. For example, the increase in stress level may be indicated by a change in the HCP's heart rate from a baseline value. The computing system can derive / infer the increase in the HCP's stress level by cross-referencing the data received from the corresponding sensing systems.

[0277] The fatigue level of a surgical technician (HCP) can be measured / determined, for example, based on instrument usage data. A computing system can calculate a weighted scale of fatigue for HCPs operating surgical instruments, as well as for other HCPs in the operating room. The weighted scale of fatigue can be based on cumulative coordinated events and contributions. For example, the weighted scale of fatigue can be based on the intensity of stress experienced by the HCP when controlling actuators such as time-dependent closure triggers, and the force applied by the HCP over time.

[0278] Details relating to the measurement of energy levels and fatigue levels are described in U.S. Patent Application No. 17 / 156,287, “METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS,” filed on 22 January 2021 (Agent Reference No. END9290USNP1), the disclosure thereof is incorporated herein by reference in its entirety.

[0279] As described herein, a computing system can adjust and / or restrict control inputs from an HCP for controlling surgical instruments. For example, a computing system can restrict control inputs from an HCP for controlling surgical instruments based on permissions such as access control levels and / or proximity to virtual boundaries of access-restricted areas such as sterile fields, operating tables, or other HCPs such as surgeons.

[0280] The computing system can adjust control inputs made by a surgical instrument controller based on authentication associated with the HCP. For example, the computing system can use authentication associated with the HCP to determine whether the HCP has permission to make control inputs and / or change the settings of the surgical instrument. Based on the authentication and / or permission, the computing system can enable or block the control inputs made by the HCP.

[0281] A computing system can restrict and / or regulate control inputs of a healthcare provider (HCP) to a device based on authentication associated with the HCP. The computing system can identify, for example, a predetermined set of devices and / or controls associated with an identified device based on authentication associated with the HCP. The computing system can monitor control inputs by the HCP to prevent, for example, inadvertent adjustment and / or control of a device that the HCP is authenticated to interact with and / or control.

[0282] For example, an anesthesiologist can have authentication to control (e.g., full control) an anesthesia device. Other HCPs can have partial control of the anesthesia device. For example, a nurse of an anesthesiologist and / or a resident of an anesthesiologist can have partial control of the anesthesia device. Based on authentication associated with the HCP, the computing system can permit or block the HCP from controlling the device. For example, the computing system can determine, based on authentication associated with an anesthesiologist, for example, that the anesthesiologist is permitted to control settings for the anesthesia device, such as sedation and / or respiratory devices, and / or have control interactions with the anesthesia device.

[0283] Based on authentication associated with other HCPs, such as an instrument nurse, the computing system can determine that the instrument nurse does not have permission to control settings and / or does not have control interactions for the anesthesia device. If the computing system determines that an HCP does not have permission to control or adjust settings for the anesthesia device, the computing system can block control inputs by the HCP and prevent inadvertent adjustment of the device by an HCP who is not authenticated to interact with and / or control the device.

[0284] The computing system can receive a disabling request from another HCP, such as a surgeon or an anesthesiologist. For example, the surgeon can disable the control inputs and / or adjustment settings of the device (e.g., disable the lockout) to distinguish, for example, between inadvertent adjustments of the device and intended or emergency adjustments.

[0285] In an example, one or more surgical instruments can be used throughout the surgical procedure. The surgical instrument may require a certain form of energy to apply a treatment and / or may have electromechanical control to operate the surgical instrument. An example of a surgical instrument can be a laparoscopic device. The surgical instrument may be used multiple times throughout the surgical procedure. The surgical instrument can be placed on / outside the operating table, passed to other HCPs, and / or disposed of after the surgical procedure.

[0286] The computing system can limit and / or adjust the control input by the HCP, for example, based on confirmation. For example, the computing system can limit and / or adjust the control input by the HCP based on visual confirmation. The computing system can detect an inadvertent control input based on visual confirmation and prevent energy from being applied to the motor power of the surgical instrument. The computing system can enable the control input if it determines that the control input is from the hand of an authorized HCP with a proper (e.g., correct) authentication having permission and / or is initiated by an authorized HCP.

[0287] As described herein, the computing system can generate a virtual boundary of an access restricted area for the target surgical zone and limit the control input by the HCP for controlling the surgical instrument. For example, the computing system can enable the control input by an authorized HCP within the virtual boundary of the target surgical zone, prevent inadvertent controls, and / or prevent damage to other HCPs or patients.

[0288] The computing system can adjust the access control level associated with an HCP based on its proximity to other HCPs, such as when a surgeon is present. For example, the computing system may determine that a scrub nurse or other personnel may not be authorized to operate a circular stapler. The computing system may determine that a scrub nurse or other personnel is near or within the virtual boundary of the surgeon's access restriction area. If the computing system determines that a scrub nurse or other personnel is near or within the virtual boundary of the surgeon's area, the computing system can adjust the access control level associated with the scrub nurse or other personnel. The computing system may enable a control input by the scrub nurse or other personnel, such as turning on the circular stapler.

[0289] The computing system can be aware of and / or identify the surgical steps of a surgical procedure. For example, the computing system can identify that the surgery is complete and / or that the surgical instruments are no longer needed for the surgery. If the computing system determines that the surgery is complete, it can monitor the activation of the surgical instruments and / or that they are powered on. If the computing system determines that the surgical instruments are activated and / or are highly energized or turned on, it can, for example, cut off the control input from the HCP to prevent accidental activation of the surgical instruments while the HCP is cleaning and / or disposing of the instruments.

[0290] If the computing system determines that the surgical procedure is complete and the surgical instrument is a battery-powered device, the computing system may initiate an energy removal process, for example, using the drain circuit of the battery-powered device. During the energy removal process, the computing system may limit (e.g., shut off) the control input from the HCP.

[0291] The computing system can adjust and / or limit the control inputs by the HCP for the gas system. During surgical procedures in the OR, medical air, oxygen, carbon dioxide, nitrogen, nitrous oxide, and / or other gases may be supplied. When the computing system detects a control input to turn the supplied gas on / off or increase / decrease it during operation, it can determine the access control level of the HCP that makes the control input to control the device supplying the gas. The computing system can determine the surgical procedure and confirm that the gas needs to be turned on / off or increased / decreased. If the computing system determines that the HCP has an access control level to adjust the gas level and that the HCP's control of the gas is consistent with the current surgical procedure, the computing system can enable the control input by the HCP. If the computing system determines that the HCP does not have an access control level and / or that the use of the gas is not consistent with the current surgical procedure, the computing system can shut off the control input by the HCP. The computing system can send alerts to the HCP as described herein.

[0292] A computing system can control the energized state of a device. The computing system can control the energized state of a device, such as a radiation device, based on the location of the HCP, the virtual boundary of the access restriction area, whether the HCP is wearing appropriate PPE, the threshold exposure time, the HCP's biomarker measurements, and the patient's biomarker measurements. As described herein, if the computing system determines that the HCP is in close proximity to the patient (e.g., within the virtual boundary of the patient, operating table, and / or surgeon) and within the surgical area, it can control the energized state of the device and enable control input from the HCP to control the device.

[0293] The computing system can monitor / determine whether an HCP, such as a radiologist or radiologic technologist, is wearing appropriate PPE. If the computing system determines that the HCP is not wearing appropriate PPE, the computing system can block the HCP's control input and send an alert. Examples of appropriate PPE may include one or more of the following, or include a lead apron, lead glass, and lead shield.

[0294] The computing system can monitor and determine whether the HCP is at an appropriate distance, such as an acceptable clearance, from the radiation device. If the computing system determines that the HCP is not within an appropriate distance, for example, within the exposure area, the computing system can shut off the control input from the HCP and send an alert.

[0295] The computing system can monitor and determine the exposure time of the HCP to radiation devices. For example, if the computing system determines that the HCP has exceeded a threshold exposure time, it can block the control input from the HCP and send an alert.

[0296] The computing system can monitor / determine biomarkers associated with the HCP or patient. For example, if the computing system determines that the HCP has an increased fatigue level and / or an increase in the level, the computing system can shut off the control input from the HCP and send an alarm. The monitoring system can monitor data associated with the patient. If the computing system determines that a biomarker associated with the patient has suddenly changed, for example, an emergency or complication has occurred, the computing system can shut off the control input from the HCP and send an alert. The computing system can determine the type of patient based on the biomarker associated with the patient. For example, the computing system can determine that the patient is a pediatric patient or a pregnant patient. To minimize radiation exposure, the computing system can shut off or adjust the control input from the HCP and send an alarm.

[0297] Figure 17 shows an exemplary flow 36450 for monitoring the movement of the HCP against the virtual boundary of an access-restricted area within the OR, as described herein. For example, in 36455, the computing system can determine the virtual boundary associated with the access-restricted area. The computing system may be or include the surgical hub 20006, the surgical hub system 20060 in Figure 4, the computer-implemented bidirectional surgical system 20070 in Figure 5, the surgical hub or computing device 20243 in Figure 7, the surgical hub 20270 in Figure 9, the console 20294 in Figure 10, and / or the surgical hub 5104 in Figure 11, as described herein with respect to Figures 1 to 3. The virtual boundaries may be, or include, the virtual boundaries associated with the restricted access area for anesthesiologists 36005, 36105, virtual boundaries associated with the restricted access area for surgeons 36025, 36115, virtual boundaries associated with the restricted access area for operating tables 36030, 36120, virtual boundaries associated with the restricted access area for scrub nurses 36035, 36125, virtual boundary associated with the restricted access area for operating room nurses 36040, virtual boundaries associated with the restricted access area for surgical instruments, sterile tray tables, and / or other surgical-related equipment 36045, 36050, 36055, 36060, 36065, virtual boundaries associated with the restricted access area for observers 36135, virtual boundary associated with the restricted access area for SSCs 36140, virtual boundaries associated with the restricted access area for assistants, and virtual boundaries associated with the restricted access area for anesthesiologist nurses, as described herein with respect to Figures 12-13.

[0298] In 36460, the computing system can identify HCPs within the OR. In 36465, the computing system can monitor the movements of the identified HCPs. For example, the computing system can determine whether an HCP is in proximity to a virtual boundary associated with an access restriction area. The computing system can determine whether an HCP is attempting to interact with a virtual boundary associated with a described access area (e.g., entering, exiting, or leaving).

[0299] In 36470, the computing system can determine whether the HCP has access authorization to interact with or enter / exit the virtual boundary associated with the restricted access area, as described herein. In 36475, the computing system can send a notice to the HCP. For example, if the computing system determines that the HCP is in proximity to the virtual boundary associated with the restricted access area (for example, based on the monitored movement of the HCP in 36465), and the computing system determines that the HCP is not authorized to interact with or enter / exit the virtual boundary associated with the restricted access area (for example, based on the HCP access authorization determination in 36470), the computing system can send a notice to the HCP indicating that the HCP is not authorized to enter the virtual boundary. If the computing system determines (for example, based on the monitored movements of the HCP in 36465) that the HCP is in proximity to a virtual boundary associated with an access-restricted area, and if the computing system determines (for example, based on the HCP access authorization determination in 36470) that the HCP is authorized to interact with or enter / exit the virtual boundary associated with the access-restricted area, the computing system may skip sending a notification to the HCP.

[0300] Figure 18 shows an exemplary flow 36500 of control access verification of HCP in OR that can be performed in the computing system described herein. The computing system may be or include the surgical hub 20006, the surgical hub system 20060 in Figure 4, the computer-implemented bidirectional surgical system 20070 in Figure 5, the surgical hub or computing device 20243 in Figure 7, the surgical hub 20270 in Figure 9, the console 20294 in Figure 10, and / or the surgical hub 5104 in Figure 11, as described herein with respect to Figures 1 to 3.

[0301] In 36505, the computing system can identify surgical instruments associated with a surgical procedure. For example, as described herein, the computing system can download a surgical plan generated by the HCP during preoperative preparation. The computing system can identify one or more surgical instruments to be used for a surgical procedure, and one or more instruments to be used for a surgical procedure.

[0302] In 36510, the computing system can detect control inputs from the HCP. For example, as described herein, the control inputs can turn a surgical instrument on / off or change the energized state of the surgical instrument (e.g., from a low energized state to a high energized state, or from a high energized state to a low energized state).

[0303] In 36515, the computing system can determine the access control level of the HCP for controlling surgical instruments. For example, the computing system can determine whether the HCP has access control authorization to control surgical instruments.

[0304] In 36520, the computing system can determine whether to enable control input by the HCP. For example, if the computing system determines that the HCP has access control authorization to control the surgical instrument, the computing system can enable the control input by the HCP to control the surgical instrument. If the computing system determines that the HCP is not authorized to control the surgical instrument (e.g., does not have access control authorization), the computing system can block the control input by the HCP.

[0305] The computing system can monitor the HCP or the device based on the position within the OR and / or the exposure time at the virtual boundary of the access-restricted area. The computing system can determine, based on local records, how long the HCP has been in the OR and / or how long the device has been in the OR. For example, the computing system can determine who has had the battery-powered device (ENDO) set up in the OR for a long time. The battery-powered device can have a 12-hour effective period once the device is set up and the battery is installed. If the computing system determines that the battery-powered device has been set up beyond the effective period, the computing system can determine that the battery-powered device may not have optimal performance.

[0306] The computing system can monitor / determine the exposure time in the OR for the device. After the device is set up, since the device can have a specific lifespan, the computing system can, for example, send an alert to the HCP to replace the device.

[0307] The computing system can track the location of devices within the OR. For example, the computing system can record the movement of devices within the OR and / or between virtual boundaries of access-restricted areas, as well as device interactions with HCPs.

[0308] The following is a non-exhaustive list of numbered embodiments that may or may not be claimable.

[0309] Embodiment 1. A computing system comprising a processor, the processor is Identify surgical instruments associated with surgical procedures in the operating room (OR), It detects control input from a medical professional (HCP) to control surgical instruments. Determine the access control level of the HCP associated with the surgical instrument, and A computing system configured to determine whether to enable a control input based on the access control level associated with the HCP, based on the control input detected by the HCP, and based on the determination that the access control level associated with the HCP does not authorize the HCP to enable the control input for controlling surgical instruments, the processor is configured to block the control input by the HCP for controlling surgical instruments, and based on the determination that the access control level associated with the HCP authorizes the HCP to enable the control input for controlling surgical instruments, the processor is configured to enable the control input by the HCP for controlling surgical instruments.

[0310] Embodiment 2. The computing system according to Embodiment 1, wherein the access control levels associated with a surgical instrument include a first access control level for controlling the surgical instrument and a second access control level for controlling the surgical instrument, the first access control level includes at least one of turning the surgical instrument on or turning the surgical instrument off, and the second access control level includes at least one of increasing the energy level associated with the surgical instrument or decreasing the energy level associated with the surgical instrument.

[0311] Embodiment 3. The control input is a first control input, the HCP is a first HCP, and the processor is The detection of a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to turn the surgical instrument on or off. To determine the access control level of the second HCP associated with the surgical instrument, The computing system according to Embodiment 1, wherein the access control level of a second HCP associated with a surgical instrument is configured to enable a second control input to turn the surgical instrument on or off based on a determination that the second HCP authorizes the surgical instrument to turn it on or off.

[0312] Embodiment 4. The control input is a first control input, the HCP is a first HCP, and the processor is Detecting a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to change the energy level associated with the surgical instrument. To determine the access control level of the second HCP associated with the surgical instrument, The computing system according to embodiment 1, wherein the access control level of a second HCP associated with a surgical instrument is configured to block a control input for controlling the surgical instrument based on a determination that the second HCP does not authorize the second HCP to change the energy level associated with the surgical instrument.

[0313] Embodiment 5. The processor is The computing system according to Embodiment 1, configured to send an alert to at least one of the devices associated with the HCP or a display in the OR, the alert comprising at least one of the following: a notification that control input by the HCP has been blocked, or access control level information assigned to the HCP.

[0314] Embodiment 6. The HCP is a first HCP, and the processor is Monitoring the movements associated with the first HCP, The first HCP is determined to be in close proximity to the operating table, Based on the determination that the first HCP is in close proximity to the operating table, an access control level adjustment inquiry message is sent to the second HCP, wherein the access control level adjustment inquiry message is configured to prompt the second HCP to indicate whether the access control level associated with the first HCP requires adjustment based on the proximity of the first HCP to the operating table. In response to the access control level adjustment inquiry message, the system receives an access control level adjustment request from the second HCP, A computing system according to embodiment 1, configured to adjust the access control level associated with a first HCP based on an access control level adjustment request.

[0315] Embodiment 7. The processor is The computing system according to aspect 6, configured to send an access control level adjustment notification to an HCP, the access control level adjustment notification indicating that the access control level associated with the HCP has been adjusted by a second HCP.

[0316] Embodiment 8. The processor is Identifying the current surgical procedure in a surgical case, Based on the identified current surgical procedure, determine whether to adjust the access control level of the HCP associated with the surgical instrument, A computing system according to embodiment 1, configured to adjust the access control level associated with the HCP, based on a determination for adjusting the access control level associated with the HCP, such that the HCP can control the surgical instruments associated with the current surgical step in a surgical procedure.

[0317] Embodiment 9. The processor is Identifying the electrical status of surgical instruments, A computing system according to Embodiment 1, configured to determine whether to adjust the access control level of an HCP associated with a surgical instrument based on the identified energized state of the surgical instrument, wherein the processor is configured to adjust the access control level of an HCP associated with the surgical instrument to block control inputs from the HCP for controlling the surgical instrument based on the identified energized state of the surgical instrument being in a high energized state, and the processor is configured to adjust the access control level of an HCP associated with the surgical instrument to enable control inputs from the HCP for controlling the surgical instrument based on the identified energized state of the surgical instrument being in a low energized state.

[0318] Embodiment 10. The processor is Monitoring biomarkers associated with HCP, wherein the biomarkers include at least one of fatigue level or stress level. A computing system according to Embodiment 1, configured to determine whether to adjust the access control level of an HCP associated with a surgical instrument based on a monitored biomarker associated with an HCP, wherein the processor is configured to adjust the access control level of an HCP associated with a surgical instrument to block control input by the HCP for controlling the surgical instrument, based on the monitored biomarker associated with the HCP indicating that the HCP has an increased level of at least one of fatigue levels or stress levels.

[0319] Embodiment 11. A method, Identifying surgical instruments associated with surgical procedures in the operating room (OR), Detecting control inputs from a healthcare professional (HCP) to control surgical instruments, To determine the access control level of HCP associated with surgical instruments, A method comprising determining whether to enable a control input based on the access control level of the HCP associated with a surgical instrument, based on a control input detected by the HCP, wherein, based on the determination that the access control level of the HCP associated with the surgical instrument does not authorize the HCP to enable a control input for controlling the surgical instrument, the control input by the HCP for controlling the surgical instrument is blocked, and based on the determination that the access control level of the HCP associated with the surgical instrument authorizes the HCP to enable a control input for controlling the surgical instrument, the control input by the HCP for controlling the surgical instrument is enabled.

[0320] Embodiment 12. The method according to Embodiment 11, wherein the access control level associated with the surgical instrument includes a first access control level for controlling the surgical instrument and a second access control level for controlling the surgical instrument, wherein the first access control level includes at least one of turning the surgical instrument on or turning the surgical instrument off, and the second access control level includes at least one of increasing the energy level associated with the surgical instrument or decreasing the energy level associated with the surgical instrument.

[0321] Embodiment 13. The control input is a first control input, the HCP is a first HCP, and the method is The detection of a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to turn the surgical instrument on or off. To determine the access control level of the second HCP associated with the surgical instrument, The method according to aspect 11, comprising enabling a second control input to turn on or off a surgical instrument, based on a determination that the access control level of a second HCP associated with the surgical instrument is authorized by the second HCP to turn the surgical instrument on or off.

[0322] Embodiment 14. The control input is a first control input, the HCP is a first HCP, and the method is Detecting a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to change the energy level associated with the surgical instrument. To determine the access control level of the second HCP associated with the surgical instrument, The method according to aspect 11, further comprising blocking a control input for controlling a surgical instrument based on a determination that the access control level of a second HCP associated with the surgical instrument does not authorize the second HCP to change the energy level associated with the surgical instrument.

[0323] Embodiment 15. The method according to Embodiment 11, comprising sending an alert to at least one of the devices associated with the HCP or a display in the OR, the alert comprising at least one of the following: a notification that control input by the HCP has been blocked, or access control level information assigned to the HCP.

[0324] Embodiment 16. HCP is a first HCP, and the method is Monitoring the movements associated with the first HCP, The first HCP is determined to be in close proximity to the operating table, Based on the determination that the first HCP is in close proximity to the operating table, an access control level adjustment inquiry message is sent to the second HCP, wherein the access control level adjustment inquiry message is configured to prompt the HCP to indicate whether the access control level of the HCP associated with the surgical instrument needs adjustment based on the proximity of the first HCP to the operating table. In response to the access control level adjustment inquiry message, the system receives an access control level adjustment request from the second HCP, The method according to aspect 11, comprising adjusting the access control level associated with a first HCP based on an access control level adjustment request.

[0325] Appearance 17. The method is: The method according to aspect 16, comprising sending an access control level adjustment notification to an HCP, the access control level adjustment notification notifying that the access control level of an HCP associated with a surgical instrument has been adjusted by a second HCP.

[0326] Apparatus 18. The method involves identifying the current surgical step in a surgical procedure, Based on the identified current surgical procedure, determine whether to adjust the access control level associated with the HCP, The method according to aspect 11, comprising adjusting the access control level of an HCP associated with a surgical instrument so that the HCP controls the surgical instrument during the current surgical step in a surgical procedure, based on a determination for adjusting the access control level of an HCP associated with the surgical instrument.

[0327] Appearance 19. The method is: Identifying the electrical status of surgical instruments, The method according to aspect 11, comprising determining whether to adjust the access control level of an HCP associated with a surgical instrument based on an identified energized state of the surgical instrument, wherein, based on the identified energized state of the surgical instrument being in a high energized state, the method includes adjusting the access control level of an HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument, and based on the identified energized state of the surgical instrument being in a low energized state, the method includes adjusting the access control level of an HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument.

[0328] Appearance 20. The method is, Monitoring biomarkers associated with HCP, wherein the biomarkers include at least one of fatigue level or stress level. The method according to aspect 11, comprising determining whether to adjust the access control level of an HCP associated with a surgical instrument based on a monitored biomarker associated with an HCP, wherein the method adjusts the access control level of an HCP associated with a surgical instrument to block control input by the HCP for controlling the surgical instrument based on the monitored biomarker associated with the HCP indicating that the HCP has an increased level of at least one of fatigue levels or stress levels, and the method adjusts the access control level of an HCP associated with a surgical instrument to enable control input by the HCP for controlling the surgical instrument based on the monitored biomarker associated with the HCP indicating that the HCP has an acceptable level of at least one of fatigue levels or stress levels.

[0329] The following is a non-exhaustive list of numbered embodiments that may or may not be claimed.

[0330] Example 1. Computer implementation method, Identifying surgical instruments associated with surgical procedures in the operating room (OR), Detecting control inputs from a healthcare professional (HCP) to control surgical instruments, To determine the access control level of HCP associated with surgical instruments, Based on the detection and control input by HCP, This includes determining whether to enable a control input based on the access control level of the HCP associated with the surgical instrument, Based on a determination that the HCP's access control level associated with the surgical instrument does not authorize the HCP to enable control inputs for controlling the surgical instrument, the HCP's control inputs for controlling the surgical instrument are blocked. A computer implementation method for enabling control inputs by an HCP to control a surgical instrument, based on a determination that the HCP's access control level associated with the surgical instrument authorizes the HCP to enable control inputs for controlling the surgical instrument.

[0331] Example 1 can provide the technical effect of preventing accidental activation of surgical instruments by authorizing or disauthorizing the HCP to enable control inputs for controlling surgical instruments. Such accidental activation of surgical instruments may occur, for example, while the HCP is cleaning and / or disposing of the instruments.

[0332] Example 2. The computer implementation method according to Example 1, wherein the access control levels associated with the surgical instrument include a first access control level for controlling the surgical instrument and a second access control level for controlling the surgical instrument, the first access control level includes at least one of turning the surgical instrument on or turning the surgical instrument off, and the second access control level includes at least one of increasing the energy level associated with the surgical instrument or decreasing the energy level associated with the surgical instrument.

[0333] Example 2 can provide the technical effect of allowing hierarchical access to surgical instruments by allowing some, but not all, controls for a specified HCP.

[0334] Example 3. The control input is the first control input, HCP is the first HCP, and the method is The detection of a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to turn the surgical instrument on or off. To determine the access control level of the second HCP associated with the surgical instrument, A computer implementation method according to Example 1 or 2, comprising enabling a second control input to turn a surgical instrument on or off based on a determination that the access control level of a second HCP associated with the surgical instrument authorizes the second HCP to turn the surgical instrument on or off.

[0335] Example 4. The control input is the first control input, the HCP is the first HCP, and the method is Detecting a second control input by a second HCP for controlling a surgical instrument, wherein the second control input is configured to change the energy level associated with the surgical instrument. To determine the access control level of the second HCP associated with the surgical instrument, A computer implementation method according to any one of Examples 1 to 3, comprising blocking a control input for controlling a surgical instrument based on a determination that the access control level of a second HCP associated with the surgical instrument does not authorize the second HCP to change the energy level associated with the surgical instrument.

[0336] Example 5. A computer implementation method according to any one of Examples 1 to 4, comprising sending an alert to at least one of the devices associated with the HCP or a display in the OR, the alert including at least one of the following: a notification that control input by the HCP has been blocked, or access control level information assigned to the HCP.

[0337] Example 5 can provide the technical benefit of allowing HCPs to remain better informed about surgical instruments within the OR and to allow HCPs to modify the access control levels of other HCPs.

[0338] Example 6. HCP is the first HCP, and the method is as follows: Monitoring the movements associated with the first HCP, The first HCP is determined to be in close proximity to the operating table, Based on the determination that the first HCP is in close proximity to the operating table, an access control level adjustment inquiry message is sent to the second HCP, wherein the access control level adjustment inquiry message is configured to prompt the HCP to indicate whether the access control level of the HCP associated with the surgical instrument needs adjustment based on the proximity of the first HCP to the operating table. In response to the access control level adjustment inquiry message, the system receives an access control level adjustment request from the second HCP, A computer implementation method according to any one of Examples 1 to 5, comprising adjusting the access control level associated with a first HCP based on an access control level adjustment request.

[0339] Example 6 can provide the technical effect of allowing more efficient surgical procedures by allowing a second HCP to supervise and adjust the access control level of the first HCP.

[0340] Example 7. The method is: The computer implementation method according to Example 6, comprising sending an access control level adjustment notification to the HCP, the access control level adjustment notification notifying that the access control level of the HCP associated with the surgical instrument has been adjusted by the second HCP.

[0341] Example 7 can provide the technical benefit of ensuring that HCPs within the OR are better informed about the actions of other HCPs, which may lead to more efficient surgical procedures.

[0342] Example 8. The method is: Identifying the current surgical procedure in a surgical case, Based on the identified current surgical procedure, determine whether to adjust the access control level associated with the HCP, A computer implementation method according to any one of Examples 1 to 7, comprising: adjusting the access control level of an HCP associated with a surgical instrument so that the HCP controls the surgical instrument during the current surgical step in a surgical procedure, based on a determination for adjusting the access control level of an HCP associated with the surgical instrument.

[0343] Example 8 can provide the technical effect of preventing or allowing the HCP to make adjustments to surgical instruments according to the current surgical procedure, thereby improving patient safety and / or surgical outcomes.

[0344] Example 9. The method is: Identifying the electrical status of surgical instruments, A computer implementation method according to any one of Examples 1 to 8, comprising: determining whether to adjust the access control level of an HCP associated with a surgical instrument based on an identified energized state of the surgical instrument; the method comprising adjusting the access control level of an HCP associated with the surgical instrument to block control input by the HCP for controlling the surgical instrument based on the identified energized state of the surgical instrument being in a high energized state; and the method comprising adjusting the access control level of an HCP associated with the surgical instrument to enable control input by the HCP for controlling the surgical instrument based on the identified energized state of the surgical instrument being in a low energized state.

[0345] Example 9 can provide a technical effect of preventing inadvertent control of surgical instruments by, for example, blocking the control input by the scrub nurse when it is determined that the surgical instrument is in a highly energized state.

[0346] Example 10. The method is: Monitoring biomarkers associated with HCP, where the biomarkers may include at least one of fatigue level or stress level. A computer implementation method according to any one of Examples 1 to 9, comprising: determining whether to adjust the access control level of an HCP associated with a surgical instrument based on a monitored biomarker associated with an HCP, wherein the method adjusts the access control level of an HCP associated with a surgical instrument to block control input by the HCP for controlling the surgical instrument based on the monitored biomarker associated with the HCP indicating that the HCP has an increased level of at least one of fatigue levels or stress levels; and the method adjusts the access control level of an HCP associated with a surgical instrument to enable control input by the HCP for controlling the surgical instrument based on the monitored biomarker associated with the HCP indicating that the HCP has an acceptable level of at least one of fatigue levels or stress levels.

[0347] Example 10 can provide the technical benefit of improving patient safety and / or surgical outcomes based on HCP biomarkers.

[0348] Example 11. A computing system, A computing system comprising a processor configured to perform the method described in any one of Examples 1 to 10.

[0349] Example 12. A data processing device comprising means for carrying out the method described in any one of Examples 1 to 10.

[0350] Example 13. A system, The computing system described in Example 11, A system comprising surgical instruments that are communicatively connected to a computer system.

[0351] Example 14. A system, The data processing device described in Example 12, A system comprising a surgical instrument communicatively connected to a data processing device.

[0352] Example 15. A computer-readable medium having instructions, wherein the instructions, when executed by a computer, enable the implementation of the method according to any one of claims 1 to 10.

[0353] [Implementation Method] (1) A computer implementation method, Identifying surgical instruments associated with surgical procedures in the operating room (OR), To detect control input from a medical professional (HCP) for controlling the aforementioned surgical instrument, Determining the access control level of the HCP associated with the surgical instrument, The process includes determining whether to enable the control input based on the access control level of the HCP associated with the surgical instrument, based on the control input detected by the HCP, Based on a determination that the access control level of the HCP associated with the surgical instrument does not authorize the HCP to enable the control input for controlling the surgical instrument, the control input by the HCP for controlling the surgical instrument is blocked. A computer implementation method for enabling the control input by the HCP for controlling the surgical instrument, based on a determination that the access control level of the HCP associated with the surgical instrument authorizes the HCP to enable the control input for controlling the surgical instrument. (2) The computer implementation method according to Embodiment 1, wherein the access control level associated with the surgical instrument includes a first access control level for controlling the surgical instrument and a second access control level for controlling the surgical instrument, the first access control level includes at least one of turning the surgical instrument on or turning the surgical instrument off, and the second access control level includes at least one of increasing the energy level associated with the surgical instrument or decreasing the energy level associated with the surgical instrument. (3) The control input is a first control input, the HCP is a first HCP, and the method is The detection of a second control input by a second HCP for controlling the surgical instrument, wherein the second control input is configured to turn the surgical instrument on or off. Determining the access control level of the second HCP associated with the surgical instrument, A computer implementation method according to Embodiment 1 or 2, comprising enabling the second control input to turn the surgical instrument on or off based on a determination that the access control level of the second HCP associated with the surgical instrument authorizes the second HCP to turn the surgical instrument on or off. (4) The control input is a first control input, the HCP is a first HCP, and the method is To detect a second control input by a second HCP for controlling the surgical instrument, wherein the second control input is configured to change the energy level associated with the surgical instrument. Determining the access control level of the second HCP associated with the surgical instrument, A computer implementation method according to Embodiment 1, comprising blocking the control input for controlling the surgical instrument based on a determination that the access control level of the second HCP associated with the surgical instrument does not authorize the second HCP to change the energy level associated with the surgical instrument. (5) The computer implementation method according to Embodiment 1, comprising sending an alert to at least one of the devices associated with the HCP or the displays in the OR, the alert comprising at least one of the following: a notification that the control input has been blocked by the HCP, or access control level information assigned to the HCP.

[0354] (6) The HCP is a first HCP, and the method is Monitoring the activity associated with the previous first HCP, The first HCP is determined to be in close proximity to the operating table, Based on the determination that the first HCP is in close proximity to the operating table, an access control level adjustment inquiry message is sent to the second HCP, wherein the access control level adjustment inquiry message is configured to prompt the HCP to indicate whether the access control level of the HCP associated with the surgical instrument needs adjustment based on the proximity of the first HCP to the operating table. In response to the aforementioned access control level adjustment inquiry message, the system receives an access control level adjustment request from the second HCP, A computer implementation method according to Embodiment 1, comprising adjusting the access control level associated with the first HCP based on the access control level adjustment request. (7) The method described above is A computer implementation method according to Embodiment 6, comprising sending an access control level adjustment notification to the HCP, the access control level adjustment notification indicating that the access control level of the HCP associated with the surgical instrument has been adjusted by the second HCP. (8) The above method is To identify the current surgical step in the aforementioned surgical procedure, Based on the identified current surgical procedure, determine whether to adjust the access control level associated with the HCP, A computer implementation method according to Embodiment 1, comprising: adjusting the access control level of the HCP associated with the surgical instrument, based on the determination for adjusting the access control level of the HCP associated with the surgical instrument, so as to enable the HCP to control the surgical instrument during the current surgical step in the surgical procedure. (9) The method described above is Identifying the energized state of the surgical instrument, A computer implementation method according to Embodiment 1, comprising determining whether to adjust the access control level of the HCP associated with the surgical instrument based on the identified energized state of the surgical instrument, wherein, based on the identified energized state of the surgical instrument being in a high energized state, the method includes adjusting the access control level of the HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument, and based on the identified energized state of the surgical instrument being in a low energized state, the method includes adjusting the access control level of the HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument. (10) The above method is Monitoring biomarkers associated with the HCP, wherein the biomarkers may include at least one of fatigue level or stress level. A computer implementation method according to Embodiment 1, comprising determining whether to adjust the access control level of the HCP associated with the surgical instrument based on a monitored biomarker associated with the HCP, wherein the method includes adjusting the access control level of the HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument based on the monitored biomarker associated with the HCP indicating that the HCP has an increased level of at least one of the fatigue level or the stress level, and adjusting the access control level of the HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument.

[0355] (11) A computing system, A computing system comprising a processor configured to perform the method described in Embodiment 1. (12) A data processing device comprising means for carrying out the method described in Embodiment 1. (13) A system, The computing system described in Embodiment 11, A system comprising a surgical instrument that is communicatively connected to the aforementioned computer system. (14) A system, The data processing device described in Embodiment 12, A system comprising a surgical instrument communicatively connected to the aforementioned data processing device. (15) A computer-readable medium having instructions, wherein the instructions, when executed by a computer, cause the method described in Embodiment 1 to be implemented.

Claims

1. A computer implementation method, Identifying surgical instruments associated with surgical procedures in the operating room (OR), To detect control input from a medical professional (HCP) for controlling the aforementioned surgical instrument, Determining the access control level of the HCP associated with the surgical instrument, The process includes determining whether to enable the control input based on the access control level of the HCP associated with the surgical instrument, based on the control input detected by the HCP, Based on a determination that the access control level of the HCP associated with the surgical instrument does not authorize the HCP to enable the control input for controlling the surgical instrument, the control input by the HCP for controlling the surgical instrument is blocked. A computer implementation method for enabling the control input by the HCP for controlling the surgical instrument, based on a determination that the access control level of the HCP associated with the surgical instrument authorizes the HCP to enable the control input for controlling the surgical instrument.

2. The computer implementation method according to claim 1, wherein the access control level associated with the surgical instrument includes a first access control level for controlling the surgical instrument and a second access control level for controlling the surgical instrument, the first access control level includes at least one of turning the surgical instrument on or turning the surgical instrument off, and the second access control level includes at least one of increasing the energy level associated with the surgical instrument or decreasing the energy level associated with the surgical instrument.

3. The control input is a first control input, the HCP is a first HCP, and the method is The detection of a second control input by a second HCP for controlling the surgical instrument, wherein the second control input is configured to turn the surgical instrument on or off. Determining the access control level of the second HCP associated with the surgical instrument, The computer implementation method according to claim 1 or 2, comprising enabling the second control input to turn the surgical instrument on or off based on a determination that the access control level of the second HCP associated with the surgical instrument authorizes the second HCP to turn the surgical instrument on or off.

4. The control input is a first control input, the HCP is a first HCP, and the method is To detect a second control input by a second HCP for controlling the surgical instrument, wherein the second control input is configured to change the energy level associated with the surgical instrument. Determining the access control level of the second HCP associated with the surgical instrument, The computer implementation method according to claim 1, comprising blocking the control input for controlling the surgical instrument based on a determination that the access control level of the second HCP associated with the surgical instrument does not authorize the second HCP to change the energy level associated with the surgical instrument.

5. The computer implementation method according to claim 1, comprising sending an alert to at least one of the devices associated with the HCP or the displays in the OR, wherein the alert includes at least one of the following: a notification that the control input has been blocked by the HCP, or access control level information assigned to the HCP.

6. The HCP is a first HCP, and the method is Monitoring the movements associated with the first HCP, The first HCP is determined to be in close proximity to the operating table, Based on the determination that the first HCP is in close proximity to the operating table, an access control level adjustment inquiry message is sent to the second HCP, wherein the access control level adjustment inquiry message is configured to prompt the HCP to indicate whether the access control level of the HCP associated with the surgical instrument needs adjustment based on the proximity of the first HCP to the operating table. In response to the aforementioned access control level adjustment inquiry message, the system receives an access control level adjustment request from the second HCP, The computer implementation method according to claim 1, comprising adjusting the access control level associated with the first HCP based on the access control level adjustment request.

7. The aforementioned method, The computer implementation method according to claim 6, comprising transmitting an access control level adjustment notification to the HCP, the access control level adjustment notification indicating that the access control level of the HCP associated with the surgical instrument has been adjusted by the second HCP.

8. The aforementioned method, To identify the current surgical step in the aforementioned surgical procedure, Based on the identified current surgical procedure, determine whether to adjust the access control level associated with the HCP, The computer implementation method according to claim 1, comprising adjusting the access control level of the HCP associated with the surgical instrument, based on the determination for adjusting the access control level of the HCP associated with the surgical instrument, such that the HCP can control the surgical instrument during the current surgical step in the surgical procedure.

9. The aforementioned method, Identifying the energized state of the surgical instrument, A computer implementation method according to claim 1, comprising determining whether to adjust the access control level of the HCP associated with the surgical instrument based on the identified energized state of the surgical instrument, wherein, based on the identified energized state of the surgical instrument being in a high energized state, the method includes adjusting the access control level of the HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument, and based on the identified energized state of the surgical instrument being in a low energized state, the method includes adjusting the access control level of the HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument.

10. The aforementioned method, Monitoring biomarkers associated with the HCP, wherein the biomarkers may include at least one of fatigue level or stress level. A computer implementation method according to claim 1, comprising determining whether to adjust the access control level of the HCP associated with the surgical instrument based on a monitored biomarker associated with the HCP, wherein the method includes adjusting the access control level of the HCP associated with the surgical instrument to block the control input by the HCP for controlling the surgical instrument, based on the monitored biomarker associated with the HCP indicating that the HCP has an increased level of at least one of the fatigue level or the stress level, and the method includes adjusting the access control level of the HCP associated with the surgical instrument to enable the control input by the HCP for controlling the surgical instrument.

11. A computing system, A computing system comprising a processor configured to carry out the method described in claim 1.

12. A data processing device comprising means for carrying out the method according to claim 1.

13. It is a system, The computing system according to claim 11, A system comprising a surgical instrument that is communicatively connected to the aforementioned computer system.

14. It is a system, A data processing device according to claim 12, A system comprising a surgical instrument communicatively connected to the aforementioned data processing device.

15. A computer-readable medium having instructions, wherein the instructions, when executed by a computer, cause the method described in claim 1 to be implemented.

Images