Dynamically determining a surgical autonomy level

Abstract

A system, method, and means for automating surgical tasks can be described herein. The device can receive instructions for surgical tasks to be performed using surgical instruments. The capabilities of the surgical instruments can be associated with a level of automation. The device can monitor the performance of the surgical task while the surgical instrument is operating at a first level of automation associated with the level of automation. The device can detect a trigger event associated with the performance of the surgical task and, for example, based on the trigger event, switch the operation of the surgical instrument from the first level of automation to a second level of automation associated with a plurality of levels of automation.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application is related to the following applications filed simultaneously, the content of each of which is incorporated herein by reference. ·U.S. Patent Application entitled "METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM" filed with this specification and having Attorney Docket No. END9430USNP1. ·U.S. Patent Application entitled "DETECTING FAILURE MITIGATION ASSOCIATED WITH AUTONOMOUS SURGICAL TASK" filed with this specification and having Attorney Docket No. END9430USNP3.

Background Art

[0002] Surgical operations are typically performed in an operating room or room within a medical facility, such as a hospital. A variety of surgical devices and systems are utilized in the performance of surgical operations. In the digital information age, medical systems and facilities are often slower to implement new and improved technologies to perform a system or procedure due to the general desire to maintain patient safety and conventional practices.

Summary of the Invention

Means for Solving the Problems

[0003] A device for changing the level of automation of a surgical instrument for performing a surgical procedure. The device comprises a processor. The processor is configured to receive instructions for a surgical procedure to be performed using the surgical instrument, the surgical procedure including a plurality of surgical tasks to be performed using the surgical instrument. The processor is configured to monitor the execution of one of the plurality of surgical tasks while the surgical instrument is operating at a first level of automation. The processor is further configured to detect a trigger event associated with the execution of the surgical task. The processor is configured to switch the operation of the surgical instrument from the first level of automation to a second level of automation based on the trigger event. Advantageously, a device for changing the level of automation of a surgical instrument for performing a surgical procedure can increase or decrease the level of automation based on the monitored execution of the surgical tasks of the procedure to reduce the risk of adverse surgical outcomes for that task or procedure, and thus ultimately improve the surgical outcome for the surgical procedure being performed. As an example, the result of a tissue resection can correspond to the integrity of the seal line, and a seal with less leakage is equivalent to a more positive result than a seal with more leakage. The result of the procedure can also correspond to the number of complications that occur during the procedure, including, for example, instrument malfunction, leakage of the seal line, misfiring of the staple line, etc.

[0004] The level of automation can correspond to the number of surgical tasks from a plurality of surgical tasks that are automated. By changing the number of automated surgical tasks, it is possible to reduce the risk to the patient and the risk of adverse surgical outcomes.

[0005] The second level of automation can be associated with automating fewer surgical tasks when compared to the first level of automation. Thus, switching the level of automation from the first level to the second level results in one or more surgical tasks that were autonomously performed before the switch and are manually performed after the switch.

[0006] Automating a surgical task can be associated with a reduction in manual input from a surgeon.

[0007] Automating a surgical task can be associated with a reduction in manual input from a surgeon.

[0008] The execution of a surgical task can be based on real-time surgical data. By monitoring the execution of a surgical task using real-time surgical data, it is possible to detect trigger events in real time and respond thereto, for example, by changing whether a monitored surgical task is automated and / or by changing whether one or more of the subsequent surgical tasks of that surgical procedure are automated, enabling the level of automation to be changed during the current surgical procedure, which can increase the likelihood of a positive surgical outcome for that surgical procedure.

[0009] The real-time surgical data may include at least one of user data, surgical environment data, surgical instrument data, and task data. For example, the surgical instrument data may include the physical state measured from a surgical device that performs a surgical task. The user data may include the measured physiological state of the patient, surgeon, and / or staff participating in the surgery.

[0010] The processor may be further configured to detect a trigger event by comparing the execution of a surgical task with a trigger event threshold.

[0011] Monitoring the execution of a surgical task may include comparing one or more of the real-time surgical data with respective ideal surgical data and detecting that a trigger event occurs when the comparison exceeds a trigger event threshold.

[0012] Each respective ideal surgical data may be generated from past data for that surgical task.

[0013] Ideal surgical data can be generated from past surgical data for surgical tasks associated with positive or ideal outcomes. If current surgical data deviates from past surgical data for surgical tasks associated with positive outcomes, it indicates that the surgical outcome for the current surgery may have a lower success rate / positivity than the outcomes associated with past surgical tasks. It may be possible to improve the likelihood of a successful surgical outcome by responding to a trigger event by changing the level of automation, which may include changing whether the current task and / or subsequent tasks are automated.

[0014] Surgery can include excising tissue. The plurality of surgical tasks may include one or more of control of an energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, reload alignment slot management.

[0015] A surgical device can include a memory that stores a plurality of surgical tasks associated with a surgery and the surgical tasks that are automated for each level of autonomy.

[0016] A surgical instrument may be suitable for performing two or more surgeries, and the surgical instrument performs a first plurality of surgical tasks for a first surgery and a second plurality of surgical tasks for a second surgery. The surgical device may include a memory that stores a first plurality of surgical tasks associated with the first surgery and a second plurality of surgical tasks associated with the second surgery. The memory may further store an association between the surgical tasks that are automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks.

[0017] A surgical instrument can be used for a plurality of procedures each including a different plurality of surgical tasks, and the device can be configured to change the level of automation of the instrument to perform any one of these procedures.

[0018] The device can be a surgical hub.

[0019] A computer-implemented method for providing commands or recommendations for changing the level of automation of surgical instruments for performing a surgical procedure is described. The method includes receiving an instruction for a surgical procedure to be performed using a surgical instrument, the surgical procedure including a plurality of surgical tasks to be performed using the surgical instrument. The method includes monitoring the execution of a surgical task among the plurality of surgical tasks while the surgical instrument is operating at a first level of automation among a plurality of levels of automation. The method further includes detecting a trigger event associated with the execution of the surgical task. The method includes providing a command to the surgical instrument or providing a recommendation to a user to switch the operation of the surgical instrument from the first level of automation to a second level of automation based on the trigger event. Advantageously, the computer-implemented method monitors the execution of surgical tasks performed by the instrument during a surgical procedure and, based on a trigger event, can provide a command to the instrument or a recommendation to the user to change the level of automation of the surgical instrument for performing the surgical procedure, increasing or decreasing the level of automation to reduce the risk of adverse surgical outcomes for that procedure and thus ultimately improving the surgical outcome for the surgical procedure being performed. As an example, the result of tissue resection can correspond to the integrity of the seal line, and a seal with less leakage is equivalent to a more positive result than a seal with more leakage. The result of the procedure can also correspond to the number of complications that occur during the surgery, including, for example, instrument malfunction, leakage of the seal line, misfiring of the staple line, etc.

[0020] The level of automation can correspond to the number of surgical tasks from a plurality of automated surgical tasks.

[0021] The second level of automation can be associated with automating fewer surgical tasks when compared to the first level of automation.

[0022] Monitoring the execution of a surgical task can be based on real-time surgical data. By using real-time surgical data to monitor the execution of a surgical task, it is possible to detect trigger events in real time and respond thereto, for example, by changing whether a monitored surgical task is automated and / or by changing whether one or more of the subsequent surgical tasks of that surgical procedure are automated, enabling the level of automation to be changed during the current surgical procedure, which can increase the likelihood of a positive surgical outcome for that surgical procedure.

[0023] Real-time surgical data may include at least one of user data, surgical environment data, surgical instrument data, and task data. For example, surgical instrument data may include a physical state measured from a surgical device performing a surgical task. User data may include a measured physiological state of a patient, surgeon, and / or staff participating in the surgery.

[0024] Monitoring the execution of a surgical task may include comparing one or more of the real-time surgical data with respective ideal surgical data and detecting that a trigger event occurs when the comparison exceeds a threshold.

[0025] Each ideal surgical data can be generated from past data about surgical tasks. Ideal surgical data can be generated from past surgical data about surgical tasks associated with positive or ideal outcomes. If the current surgical data deviates from past surgical data about surgical tasks associated with positive outcomes, it indicates that the surgical outcome for the current surgery may have a lower success rate / positivity than the outcomes associated with past surgical tasks. It may be possible to improve the likelihood of a successful surgical outcome by responding to trigger events by changing the level of automation, which may include changing whether the current task and / or subsequent tasks are automated.

[0026] The method may further include generating each ideal surgical data.

[0027] The method may include retrieving from memory a plurality of surgical tasks associated with a surgical procedure and the surgical tasks automated for each level of autonomy.

[0028] The surgical instrument may be suitable for performing two or more surgical procedures, and the surgical instrument performs a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure. The method may further include retrieving from memory instructions for a first plurality of surgical tasks associated with a first surgical procedure and a second plurality of surgical tasks associated with a second surgical procedure, and retrieving from memory an association between the surgical tasks automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks.

[0029] The surgical instrument can be used for a plurality of procedures each including a different plurality of surgical tasks, and the device can be configured to change the level of automation of the instrument to perform any one of these procedures.

[0030] A computer program comprising instructions which, when the program is executed by a computer, cause the computer to perform any of the aforementioned methods.

[0031] A computer-readable medium comprising instructions which, when executed by a controller, cause the controller to perform any of the aforementioned methods.

[0032] A device for performing a plurality of surgical tasks in a surgical procedure is described. The device comprises a processor. The processor is configured to operate at a first level of automation. The processor is configured to receive an instruction to switch to a second level of automation, the instruction being based on detecting a trigger. The processor is further configured to operate at the second level of automation based on the instruction. Advantageously, a device that can operate at different levels of automation allows for changes in automation and allows the procedure to be more or less automated, reducing the risk of adverse surgical outcomes for that procedure and thus ultimately improving the surgical outcome for the surgical procedure being performed.

[0033] The first level of automation and the second level of automation may differ in the number of surgical tasks of the surgical procedure being automated.

[0034] The instruction may be based on monitoring the execution of one of the plurality of surgical tasks at the first level of automation.

[0035] The execution of the surgical tasks at the first level of automation can be based on real-time surgical data.

[0036] By monitoring the execution of surgical tasks using real-time surgical data, it is possible to detect trigger events in real time and respond thereto, for example, by changing whether the monitored surgical task is automated and / or by changing whether one or more of the subsequent surgical tasks of the surgery are automated, enabling the level of automation to be changed during the current surgery, which can increase the likelihood of a positive surgical outcome for that surgery.

[0037] Surgical tasks can be automated. The device can receive instructions for surgical tasks performed using surgical instruments. The capabilities of the surgical instruments can be associated with the level of automation. The device can monitor the execution of surgical tasks while the surgical instruments are operating at a first level of automation associated with the level of automation. The device can detect a trigger event associated with the execution of the surgical task and, for example, based on the trigger event, switch the operation of the surgical instruments from the first level of automation to a second level of automation associated with a plurality of levels of automation. The capabilities of the surgical instruments can include a set of surgical instrument tasks, and the level of automation can be associated with automating one or more surgical instrument tasks from the set of surgical instrument tasks. The execution of the surgical task can be based on real-time surgical data. The real-time surgical data can include one or more of user data, surgical environment data, surgical instrument data, task data, past data, etc. In an example, the device can detect a trigger event by comparing the execution of the surgical task to a trigger event threshold and, based on the trigger event, switch the operation of the surgical instruments from the first level of automation to the second level of automation. The second level of automation can be associated with automating fewer surgical instrument tasks when compared to the first level of automation. Monitoring the execution of the surgical task can include comparing one or more of the real-time surgical data to respective ideal surgical data.

[0038] In an example, the device may operate at a first level of automation among levels of automation associated with a surgical task. The device may obtain an instruction to switch to a second level of automation among levels of automation associated with the surgical task. The instruction may be based on the detection of a trigger. The device may operate at a second level of automation among levels of automation associated with the surgical task based on the instruction. The instruction may be based on monitoring the execution of the first level of automation. The execution of the first level of automation may be based on real-time surgical data.

Brief Description of the Drawings

[0039]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0040] FIG. 1 is a block diagram of a computer-implemented surgical system 20000. An exemplary surgical system such as surgical system 20000 can include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, each surgical system 20002 can include a computer-implemented bidirectional surgical system. For example, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 that communicate with a cloud computing system 20008, as described, for example, in FIG. 2. The cloud computing system 20008 can include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Exemplary surgical systems 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include, for example, one or more devices used to measure one or more environmental attributes, as further described, for example, in FIG. 2. The robotic system 20013 may include, for example, a plurality of devices used to perform a surgical procedure, as further described, for example, in FIG. 2.

[0041] The surgical system 20002 can communicate with a remote server 20009 that can be part of a cloud computing system 20008. In one example, the surgical system 20002 may communicate with the remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, the patient sensing system may communicate directly with the remote server 20009. The surgical system 20002 and / or its components may use one or more of the cellular protocols of GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A), new radio (NR) or 5G to communicate with the remote server 20009 via a cellular transmission / reception point (TRP) or base station.

[0042] The surgical hub 20006 can have a collaborative interaction with one of more means for displaying images from the laparoscope scope as well as information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from one or more sensing systems 20011 and transmit notification or control messages to one or more sensing systems 20011. The surgical hub 20006 can transmit and / or receive information including notification information to and from a human interface system 20012. The human interface system 20012 can include one or more human interface devices (HID). The surgical hub 20006 can transmit and / or receive acoustic devices, notification or control information to a display, and / or control information to various devices that communicate with the surgical hub.

[0043] For example, as discussed in FIG. 1, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and one or more patient sensing systems), and an environmental sensing system 20015. One or more sensing systems 20001 may measure data regarding various biomarkers. One or more sensing systems 20001 may use one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc., to measure the biomarkers. One or more sensors may use one of more of the sensing techniques such as photoplethysmography, electrocardiogram examination, electroencephalogram examination, colorimetric analysis, impedancemetry, potential difference measurement, current measurement, etc., to measure the biomarkers as described herein.

[0044] The biomarkers measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximum oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood potential of hydrogen, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous tension, circadian rhythm, and / or menstrual cycle.

[0045] Biomarkers can be related to physiological systems, including but not limited to the behavioral and psychological, cardiovascular, renal, skin, nervous, gastrointestinal, respiratory, endocrine, immune, tumor, musculoskeletal, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgical system 20000. Information from biomarkers may be determined and / or used by the computer-implemented patient and surgical system 20000 to, for example, improve the above system and / or improve patient outcomes. One or more sensing systems 20001, biomarkers 20005, and physiological systems are described in more detail by U.S. Patent Application No. 17 / 156,287, filed January 22, 2021, entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS" (Attorney Docket No. END9290USNP1), the disclosure of which is incorporated herein by reference in its entirety.

[0046] Figure 2 shows an example of a surgical system 20002 in an operating room. As illustrated in Figure 2, the patient is operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more HCP sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding the HCPs may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors deployed in the operating room. The HCP sensing system 20020 and the environmental sensing system communicate with the surgical hub 20006 and may further communicate with one or more cloud servers 20009 of the cloud computing system 20008, as shown in Figure 1. The environmental sensing system may be used to measure one or more environmental attributes, such as the location of HCPs in the operating room, HCP movement, ambient noise in the operating room, temperature / humidity in the operating room, etc.

[0047] As illustrated in FIG. 2, the main display 20023 and one or more audio output devices (e.g., speaker 20019) are placed in the sterile field so as to be visible to the operator on the operating table 20024. In addition, the visualization / notification tower 20026 is placed outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human interface device (HID) 20027 and a second non-sterile HID 20029 facing opposite each other. The HID may be a display or a display having a touch screen that enables a human to directly interface with the HID. The human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to regulate the information flow to the operators inside and outside the sterile field. In one example, by the surgical hub 20006, the HID (e.g., the main HID 20023) may display notifications and / or information regarding the patient and / or surgical steps. In one example, the surgical hub 20006 may prompt and / or receive input from a person within the sterile field or non-sterile area. In one example, by the surgical hub 20006, the HID may display a snapshot of the surgical site recorded by the imaging device 20030 on the non-sterile HID 20027 or 20029 while maintaining a live video of the surgical site on the main HID 20023. The snapshot on the non-sterile display 20027 or 20029 may, for example, permit a non-sterile operator to perform diagnostic steps related to the surgery.

[0048] In one aspect, the surgical hub 20006 may be configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to the main display 20023 within the sterile field so that it can be seen by the sterile operator on the operating table. In one example, the input may be in the form of a modification to a snapshot displayed on the non-sterile display 20027 or 20029 that can be sent by the surgical hub 20006 to the main display 20023.

[0049] Referring to FIG. 2, the surgical instrument 20031 is used as part of a surgical system 20002 in a surgical operation. The hub 20006 can also be configured to regulate the information flow to the display of the surgical instrument 20031. For example, in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 20026 are sent by the hub 20006 to the surgical instrument display within the sterile field, where they can be viewed by the operator of the surgical instrument 20031. Exemplary surgical instruments suitable for use with the surgical system 20002 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0050] FIG. 2 illustrates an example of a surgical system 20002 used to perform surgery on a patient lying on an operating table 20024 within an operating room 20035. A robotic system 20034 can be used as part of the surgical system 20002 in a surgical procedure. The robotic system 20034 can include a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. While the surgeon views the surgical site through the surgeon's console 20036, the patient-side cart 20032 can manipulate at least one removably coupled surgical tool 20037 through a minimally invasive incision in the patient's body. Images of the surgical site can be acquired by a medical imaging device 20030 that can be manipulated by the patient-side cart 20032 to change the orientation of the imaging device 20030. The robotic hub 20033 can be used to process the images of the surgical site and then display them to the surgeon through the surgeon's console 20036.

[0051] Other types of robotic systems can be readily adapted for use with the surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019 / 0201137 (A1) (U.S. Patent Application No. 16 / 209,407), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed on Dec. 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0052] Various examples of cloud-based analysis methods implemented by a cloud computing system 20008 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569 (A1) (U.S. Patent Application No. 16 / 209,403), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed on Dec. 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0053] In various aspects, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0054] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may be capable of receiving light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0055] The one or more illumination sources may be configured to emit electromagnetic energy within the visible spectrum as well as the invisible spectrum. The visible spectrum is sometimes also referred to as the optical spectrum or emission spectrum and is a portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye) and may be referred to as visible light or simply light. Typically, the human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0056] The invisible spectrum (e.g., non-emission spectrum) is a portion of the electromagnetic spectrum that is located below and above the visible spectrum (i.e., wavelengths less than approximately 380 nm and greater than approximately 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet, x-rays, and gamma ray electromagnetic radiation.

[0057] In various aspects, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0058] The imaging device may employ multispectral monitoring to distinguish topography from the underlying structure. A multispectral image captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments with sensitivity to specific wavelengths including frequencies beyond the visible light range, e.g., IR, and light from ultraviolet. Spectral imaging enables extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is detailed under the heading “Advanced Imaging Acquisition Module” in U.S. Patent Application Publication No. 2019 / 0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed Dec. 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after a surgical task for performing one or more of the above-described tests on the treated tissue has been completed. It is understood that strict sterilization of the operating room and surgical instruments is required during any surgical procedure. The strict hygiene and sterilization conditions required in the “operating room,” i.e., the operating or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize anything that comes into contact with the patient or enters the sterile field, including the imaging device 20030 and its accessories and components. It will be understood that the sterile field can be considered a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding a patient prepared for surgery. The sterile field can include properly attired and scrubbed team members, as well as all equipment and fixtures within that area.

[0059] The wearable sensing system 20011 shown in FIG. 1 may include one or more sensing systems, such as the HCP sensing system 20020 as shown in FIG. 2. The HCP sensing system 20020 may include a sensing system for monitoring and detecting a set of physical states and / or a set of physiological states of a healthcare provider (HCP). The HCP may generally be one or more healthcare providers who assist a surgeon or other healthcare service provider. In one example, the sensing system 20020 may measure a set of biomarkers to monitor the heart rate of the HCP. In one example, the sensing system 20020 (e.g., a watch or a wristband) worn on the wrist of a surgeon may use an accelerometer to detect hand movement and / or shake, and determine the magnitude and frequency of tremors. The sensing system 20020 may transmit measurement data associated with the set of biomarkers and data associated with the physical state of the surgeon to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the position of the HCP's hand / body. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include devices such as a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the operating room. The surgical hub 20006 may, alone or in communication with a cloud computing system, use the surgeon biomarker measurement data and / or environmental sensing information to, for example, modify the control algorithm of a handheld instrument or the average latency of a robotic interface to minimize tremors. In one example, the HCP sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit measurement data associated with the surgeon biomarkers to the surgical hub 20006.The HCP perception system 20020 may use one or more of the RF protocols of Bluetooth (registered trademark), Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 low-power wireless personal area network (6LoWPAN), and Wi-Fi to communicate with the surgical hub 20006. The surgeon biomarkers may include one or more of stress, heart rate, etc. The environmental measurement values from the operating room may include the ambient noise level related to the movement of the surgeon or patient, surgeon and / or staff, the attention level of the surgeon and / or staff, etc.

[0060] The surgical hub 20006 may adaptively control one or more surgical instruments 20031 using the surgeon biomarker measurement data associated with the HCP. For example, the surgical hub 20006 may send a control program to the surgical instrument 20031 to control its actuator to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may send the control program based on situation recognition and / or the importance or criticality of the task. The control program may instruct the instrument to change its operation to provide more control when control is needed.

[0061] Figure 3 shows an exemplary surgical system 20002 having a surgical hub 20006. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014 via a modular control unit. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain aspects, as illustrated in Figure 3, the hub 20006 further includes an exhaust smoke module 20054 and / or a suction / perfusion module 20055. The various modules and systems can be connected directly or via the communication module 20056 to the modular control unit via a router. The operating room devices can be coupled to cloud computing resources and data storage via the modular control unit. The human interface system 20012 can include a display subsystem and a notification subsystem.

[0062] The modular control unit may be connected to a non-contact sensor module. The non-contact sensor module may use ultrasonic, laser type, and / or similar non-contact measurement devices to measure the dimensions of the operating room and generate a map of the operating room. Other distance sensors can be used to determine the boundaries of the operating room. In U.S. Provisional Patent Application No. 62 / 611,341, filed on December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM", which is hereby incorporated by reference in its entirety, as described under the heading "Surgical Hub Spatial Awareness Within an Operating Room" in the same document, an ultrasonic-based non-contact sensor module can scan the operating room by transmitting an ultrasonic burst and receiving the echo when the ultrasonic burst is reflected from the outer wall of the operating room. The sensor module may be configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. A laser-based non-contact sensor module can scan the operating room, for example, by transmitting a laser light pulse, receiving the laser light pulse reflected from the outer wall of the operating room, comparing the phase of the transmitted pulse with the received pulse to determine the size of the operating room, and adjusting the Bluetooth pairing distance limit.

[0063] During surgery, applying energy to tissue for sealing and / or cutting is generally associated with smoke evacuation, aspiration of excess fluid, and / or perfusion of tissue. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgery. Valuable time can be lost in dealing with this problem during surgery. To untangle the lines, it may be necessary to unplug them from their corresponding modules, which may require resetting the modules. The hub module type enclosure 20060 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglements. Aspects of the present disclosure present a surgical hub 20006 for use in surgeries involving applying energy to tissue at a surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combined generator module slidably receivable within a docking station of the hub enclosure 20060. The docking station includes data contacts and power contacts. The combined generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit. In one aspect, the combined generator module also includes a smoke evacuation component, at least one energy supply cable for connecting the combined generator module to a surgical instrument, at least one smoke evacuation component configured to discharge smoke, fluid, and / or particulates generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke evacuation component. In one aspect, the fluid line may be a first fluid line, and a second fluid line may extend from a remote surgical site to a suction and perfusion module 20055 slidably receivable within the hub enclosure 20060. In one aspect, the hub enclosure 20060 may include a fluid interface. Certain surgeries may require applying two or more energy types to tissue.One type of energy may be more beneficial for cutting tissue, while a different type of energy may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut the sealed tissue. Aspects of the present disclosure present a solution where a hub module enclosure 20060 is configured to house different generators and facilitate two-way communication between them. One advantage of the hub module enclosure 20060 is that it allows for the quick removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port that includes first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is also slidably movable to disengage from the electrical engagement with the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue that is different from the first energy, and a second docking station having a second docking port that includes second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is also slidably movable to disengage from the electrical engagement with the second power and data contacts. Additionally, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port configured to facilitate communication between the first energy generator module and the second energy generator module.Referring to FIG. 3, aspects of the present disclosure are presented regarding a generator module 20050, a smoke exhaust module 20054, and a hub module enclosure 20060 that enables modular integration of the suction / irrigation module 20055. The hub module enclosure 20060 further facilitates two-way communication between the module 20059, the module 20054, and the module 20055. The generator module 20050 may include integrated monopolar components, bipolar components, and ultrasonic components supported within a single housing unit slidably insertable into the hub's modular enclosure 20060. The generator module 20050 may be configured to connect to a monopolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the hub module enclosure 20060. The hub module enclosure 20060 can be configured to facilitate the insertion of multiple generators and two-way communication between the generators docked to the hub module enclosure 20060 such that the multiple generators function as a single generator.

[0064] FIG. 4 illustrates a surgical data network having a set of communication hubs configured to connect to a cloud, a set of sensing systems, an environmental sensing system, and a set of other modular devices disposed in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specially equipped for surgery within a medical facility, according to at least one aspect of the present disclosure.

[0065] As illustrated in FIG. 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices disposed within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 that may include a remote server 20067 connected to a remote storage 20068). The modular communication hub 20065 and the devices may be connected in a room within a medical facility that is specially equipped for surgical procedures. In one aspect, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicate with a network router 20066. The modular communication hub 20065 may also be coupled to a local computer system 20063 and provide local computer processing and data manipulation.

[0066] The computer system 20063 may include a processor and a network interface 20100. The processor may be connected via a system bus to a communication module, storage, memory, non-volatile memory, and an input / output (I / O) interface. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures, examples of which include a 9-bit bus, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus, but is not limited thereto.

[0067] The processor may be any single-core or multi-core processor, such as those known by the product name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core has on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle serial random access memory (SRAM), an internal read-only memory (ROM) with StellarisWare® software, 2KB of electrically erasable programmable read-only memory (EEPROM) and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, the details of which are available in the product datasheet.

[0068] In one example, the processor may include a safety controller with two controller-based families such as TMS570 and RM4x, also known by the product name Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, while providing scalable performance, connectivity, and memory options and providing a high level of integrated safety mechanisms.

[0069] It should be understood that the computer system 20063 may include software that functions as a medium between the described user and the basic computer resources in a suitable operating environment. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may utilize the resource management by the operating system via program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0070] A user can input commands or information into the computer system 20063 via an input device connected to the I / O interface. Examples of input devices include, but are not limited to, pointing devices such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, gamepad, satellite broadcast receiving antenna, scanner, TV tuner card, digital camera, digital video camera, web camera, etc. These and other input devices are connected to the processor 20102 through the system bus via an interface port. Examples of interface ports include, for example, serial ports, parallel ports, game ports, and USB. The output device uses some of the same type of ports as the input device. Thus, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to the output device. The output adapter may be provided to illustrate that among the output devices that may require a special adapter, there can be several output devices such as a monitor, display, speaker, and printer. Examples of output adapters include, but are not limited to, video and sound cards that provide connection means between the output device and the system bus. It should be noted that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0071] The computer system 20063 can operate in a networked environment that uses logical connections to one or more remote computers, such as a cloud computer, or a local computer. The remote cloud computer can be, for example, a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, or other common network nodes, but typically includes many or all of the elements described with respect to the computer system. For simplicity, only a memory storage device is illustrated together with the remote computer. The remote computer can be logically connected to the computer system via a network interface and subsequently physically connected via a communication connection. The network interface can include communication networks such as a local area network (LAN) and a wide area network (WAN). Examples of LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE802.3, Token Ring / IEEE802.5, etc. Examples of WAN technologies include circuit-switched networks such as point-to-point links, Integrated Services Digital Network (ISDN) and its variants, packet-switched networks, and Digital Subscriber Line (DSL), but are not limited thereto.

[0072] In various examples, computer system 20063 may include an image processor, an image processing engine, a media processor, or any special digital signal processor (DSP) used for processing digital images. The image processor can enhance speed and efficiency using parallel computing with single instruction, multiple data (SIMD), or multiple instruction, multiple data (MIMD) techniques. The digital image processing engine can perform various tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0073] The communication connection part may refer to the hardware / software used to connect a network interface to a bus. For exemplary clarity, the communication connection part is shown inside computer system 20063, but the communication connection part may be outside computer system 20063. For illustrative purposes only, the hardware / software required for connection to a network interface can include modems such as ordinary telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and internal and external technologies such as Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0074] The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switching. A passive surgical data network functions as a conduit for data, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through the surgical data network being monitored and includes additional features that configure each port within network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0075] The modular devices 1a to 1n arranged in the operating room can be connected to the modular communication hub 20065. The network hub 20061 and / or the network switch 20062 can be connected to the network router 20066 to connect the devices 1a to 1n to the cloud computing system 20064 or the local computer system 20063. The data associated with the devices 1a to 1n may be transferred via the router to a cloud-based computer for remote data processing and operation. The data associated with the devices 1a to 1n can also be transferred to the local computer system 20063 for local data processing and operation. The modular devices 2a to 2m arranged in the same operating room may also be connected to the network switch 20062. The network switch 20062 can be connected to the network hub 20061 and / or the network router 20066 to connect the devices 2a to 2m to the cloud 20064. The data associated with the devices 2a to 2m can be transferred via the network router 20066 to the cloud computing system 20064 for data processing and operation. The data associated with the devices 2a to 2m may also be transferred to the local computer system 20063 for local data processing and operation.

[0076] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing system 20069 may include an HCP sensing system and / or a patient sensing system. One or more sensing systems 20069 may communicate with the computer system 20063 of the surgical hub system 20060 or the cloud server 20067 directly via one of the network routers 20066 or via the network hub 20061 or the network switching 20062 that communicates with the network router 20066.

[0077] The sensing system 20069 can be coupled to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 can be transferred via the network router 20066 to the cloud computing system 20064 for data processing and manipulation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0078] As illustrated in FIG. 4, the surgical hub system 20060 can be expanded by interconnecting a plurality of network hubs 20061 and / or a plurality of network switches 20062 with a plurality of network routers 20066. The modular communication hub 20065 can be housed within a modular control tower configured to receive a plurality of devices 1a - 1n / 2a - 2m. The local computer system 20063 may also be housed within the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images obtained by some of the devices 1a - 1n / 2a - 2m, for example, during a surgical procedure. In various aspects, the devices 1a - 1n / 2a - 2m can include various modules such as, among other modular devices that can be connected to the modular communication hub 20065 of a surgical data network, an imaging module coupled to an endoscope, a generator module coupled to an energy-based surgical device, a smoke evacuation module, a suction / irrigation module, a communication module, a processor module, a storage array, a surgical device coupled to a display, and / or a non-contact sensor module.

[0079] In one aspect, the surgical hub system 20060 illustrated in FIG. 4 may include a combination of a network hub, a network switch, and a network router that connect devices 1a-1n / 2a-2m or the sensing system 20069 to the cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or the sensing system 20069 connected to the network hub 20061 or the network switch 20062 may collect data in real time and transfer the data to a cloud computer for data processing and operation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term "cloud" may be used as a metaphor for the "Internet," but this term is not so limited. Thus, the term "cloud computing" can be used herein to refer to "one type of Internet-based computing," in which case various services such as servers, storage, and applications are distributed via the Internet to the modular communication hub 20065 and / or the computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or the computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of the devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services may perform a number of calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0080] By applying cloud computing data processing technology to the data collected by devices 1a to 1n / 2a to 2m, the surgical data network can provide improvements in surgical outcomes, cost reduction, and patient satisfaction. After tissue sealing and cutting procedures, at least some of devices 1a to 1n / 2a to 2m can be used to observe the tissue state to evaluate leakage or perfusion of the sealed tissue. At least some of devices 1a to 1n / 2a to 2m can be used to examine data including images of samples of body tissues for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of diseases. This can include tissue localization, margin confirmation, and phenotype. At least some of devices 1a to 1n / 2a to 2m can be used to identify the anatomical structures of the body using various sensors integrated with the imaging device and techniques such as overlaying images captured by multiple imaging devices. The data collected by devices 1a to 1n / 2a to 2m, including image data, can be transferred to cloud computing system 20064 or local computer system 20063 or both for data processing and operations including image processing and manipulation. The data may be analyzed to improve the results of surgery by determining whether further treatments such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and precision robotics can be performed on tissue-specific sites and conditions. Such data analysis may further employ prognostic analysis processing, and using standardized methods can provide useful feedback either to confirm surgical treatment and surgeon behavior or to propose modifications to surgical treatment and surgeon behavior.

[0081] When cloud computer data processing technology is applied to the measurement data collected by the sensing system 20069, the surgical data network can result in improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing system 20069 may be used to evaluate the physiological state of a surgeon performing a surgery, a patient being prepared for surgery, or a patient recovering after surgery. The cloud-based computing system 20064 can monitor biomarkers associated with a surgeon or patient in real time, generate a surgical plan based at least on the measurement data collected before surgery, supply control signals to surgical instruments during surgery, and be used to notify a patient of complications during the postoperative period.

[0082] The operating room devices 1a to 1n can be connected to the modular communication hub 20065 via a wired channel or a wireless channel according to the configuration of the devices 1a to 1n with respect to the network hub 20061. In one aspect, the network hub 20061 may be implemented as a local network broadcast device that functions on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a to 1n located within the same operating room network. The network hub 20061 can collect data in the form of packets and transmit them to the router in half-duplex mode. The network hub 20061 cannot store any media access control / Internet Protocol (MAC / IP) for transferring device data. Only one of the devices 1a to 1n can transmit data at a time via the network hub 20061. The network hub 20061 cannot have a routing table or intelligence regarding the destination of information and broadcasts all network data across each connection and to the remote server 20067 of the cloud computing system 20064. The network hub 20061 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose a security risk and cause a bottleneck.

[0083] The operating room devices 2a - 2m can be connected to the network switch 20062 via a wired channel or a wireless channel. The network switch 20062 functions within the data link layer of the OSI model. The network switch 20062 may be a multicast device for connecting the devices 2a - 2m arranged in the same operating room to the network. The network switch 20062 transmits data in the form of frames to the network router 20066 and can function in full - duplex mode. Multiple devices 2a - 2m can transmit data simultaneously via the network switch 20062. The network switch 20062 stores and uses the MAC addresses of the devices 2a - 2m for data transfer.

[0084] The network hub 20061 and / or the network switch 20062 can be connected to the network router 20066 for connecting to the cloud computing system 20064. The network router 20066 functions within the network layer of the OSI model. The network router 20066 creates a route for transmitting the data packets received from the network hub 20061 and / or the network switch 20062 to the cloud - based computer resources for further processing and operation of the data collected by any one or all of the devices 1a - 1n / 2a - 2m and the wearable sensing system 20011. The network router 20066 may be used to connect two or more different networks located at different positions, such as different operating rooms in the same medical facility or different networks in different operating rooms of different medical facilities. The network router 20066 transmits data in the form of packets to the cloud computing system 20064 and can function in full - duplex mode. Multiple devices can transmit data simultaneously. The network router 20066 can use IP addresses for data transfer.

[0085] In one example, the network hub 20061 may be implemented as a USB hub that enables a plurality of USB devices to be connected to a host computer. The USB hub can expand a single USB port into several tiers so that there are more available ports for connecting devices to the host system computer. The network hub 20061 may include a wired function or a wireless function for receiving information via a wired channel or a wireless channel. In one aspect, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a to 1n and devices 2a to 2m located within an operating room.

[0086] In an example, the operating room devices 1a-1n / 2a-2m and / or the sensing system 20069 may communicate with the modular communication hub 20065 via the Bluetooth wireless technology standard to exchange data over a short distance from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and to construct a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or the sensing system 20069 may communicate with the modular communication hub 20065 via some wireless communication standards or wired communication standards or protocols, such as Bluetooth, Low-Energy Bluetooth, near-field communication (NFC), Wi-Fi (IEEE802.11 family), WiMAX (IEEE802.16 family), IEEE802.20, New Radio (NR), Long Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, as well as 3G, 4G, 5G, and any other wireless protocol and wired protocol specified in the future, but not limited thereto. The computing module may include a plurality of communication modules. For example, the first communication module may be dedicated to short-distance wireless communication such as Wi-Fi and Bluetooth, Low-Energy Bluetooth, Bluetooth Smart, etc., and the second communication module may be dedicated to long-distance wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, etc.

[0087] The modular communication hub 20065 functions as a central connection for one or more of the operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 and can handle data types known as frames. The frames can carry data generated by the devices 1a - 1n / 2a - 2m and / or the sensing system 20069. When a frame is received by the modular communication hub 20065, the frame is amplified and sent to the network router 20066, which can transfer this data to the cloud computing system 20064 or the local computer system 20063 by using a number of wireless communication standards or wired communication standards or protocols as described herein.

[0088] The modular communication hub 20065 may be used as a stand - alone device or may be connected to compatible network hubs 20061 and network switches 20062 to form a larger network. The modular communication hub 20065 can be a good option for networking the operating room devices 1a - 1n / 2a - 2m as it is generally easy to install, configure, and maintain.

[0089] FIG. 5 illustrates a logic diagram of a control system 20220 for a surgical instrument or a surgical tool according to one or more aspects of the present disclosure. The surgical instrument or the surgical tool may be configurable. The surgical instrument may be at hand, such as an imaging device, a surgical stapler, an energy device, an end cutter device, and may include surgical supplies specific to the procedure. For example, the surgical instrument may include any one of an electric stapler, an electric stapler generator, an energy device, a high energy device, a high energy Joe device, an end cutter clamp, an energy device generator, an in - operating room imaging system, a smoke evacuation device, a suction irrigation device, a pneumoperitoneum system, etc. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221 including a processor 20222 and a memory 20223. For example, one or more of sensors 20225, 20226, 20227 provide real - time feedback to the processor 20222. A motor 20230 driven by a motor driver 20229 is operably coupled to a longitudinally movable displacement member to drive an I - beam knife element. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. The position information may be provided to a processor 20222 that may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the position of the firing member, the firing bar, and the I - beam knife element. Additional motors may be provided to the tool driver interface to control the firing of the I - beam, the movement of the closure tube, the rotation of the shaft, and the articulation movement. A display 20224 may display various operating states of the instrument and may include a touch - screen function for data input. The information displayed on the display 20224 may be overlaid with an image acquired via an endoscope imaging module.

[0090] The microcontroller 20221 may be any single-core or multi-core processor, such as those known by the trade names of ARM Cortex from Texas Instruments. In one aspect, the main microcontroller 20221 may be, for example, an on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum of 40MHz, the details of which are available in the product datasheet, a prefetch buffer for improving performance beyond 40MHz, 32KB single-cycle SRAM, an internal ROM with StellarisWare (registered trademark) software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels, which may be the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments.

[0091] The microcontroller 20221 may also include a safety controller with two controller-based families such as TMS570 and RM4x, known by the trade names of Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0092] The microcontroller 20221 may be programmed to perform various functions such as precise control over the speed and position of the knife and the articulation movement system. In one aspect, the microcontroller 20221 may include a processor 20222 and a memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the articulation movement section or the knife system. In one aspect, the motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be readily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT", published on October 19, 2017, which is hereby incorporated by reference in its entirety.

[0093] The microcontroller 20221 may be programmed to provide accurate control over the speed and position of the displacement member and the articulation movement system. The microcontroller 20221 may be configured to calculate responses within the software of the microcontroller 20221. The calculated responses can be compared with the measured responses of the actual system to obtain an "observed" response, which is used for actual feedback decision-making. The observed response may be a suitably adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0094] The motor 20230 may be controlled by a motor driver 20229 and can also be used by a surgical instrument or a tool firing system. In various forms, the motor 20230 may be a brushed DC drive motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver with field effect transistors (FETs). The motor 20230 may be powered by a power supply assembly removably attached to a handle assembly or a tool housing to supply control power to the surgical instrument or tool. The power supply assembly may include a battery that may include a number of battery cells connected in series that can be used as a power source to power the surgical instrument or tool. In certain situations, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be lithium-ion batteries that can be connectable to and separable from the power supply assembly.

[0095] The motor driver 20229 may be the A3941 available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include a proprietary charge pump regulator, which supplies a full (>10V) gate drive to battery voltages up to 7V, enabling the A3941 to operate with a reduced gate drive up to 5.5V. A bootstrap capacitor may be used to supply the battery supply voltage required for the N-channel MOSFET. The internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible through either the high-side FET or the low-side FET. The power FETs can be protected from shoot-through by a resistor-adjustable dead time. The integrated diagnostics indicate low voltage, overtemperature, and power bridge anomalies and can be configured to protect the power MOSFETs under most short-circuit conditions. Other motor drivers may be easily substituted for use in the tracking system 20228 with an absolute positioning system.

[0096] Tracking system 20228 may comprise a controlled motor drive circuit configuration with a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for an absolute positioning system may supply a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member having a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reduction assembly. In some examples, the displacement member may represent a firing member adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a firing bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, a firing member, a firing bar, an I-beam, or any element that may be displaced. In one aspect, the longitudinally movable drive member may be coupled to the firing member, the firing bar, and the I-beam. Thus, the absolute positioning system may actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member may be coupled to any position sensor 20225 suitable for measuring linear displacement. Thus, the longitudinally movable drive member, the firing member, the firing bar, or the I-beam, or combinations thereof, may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor.Linear displacement sensors may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system including a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system including a fixed magnet and a series of linearly arranged movable Hall effect sensors, an optical detection system including a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system including a fixed light source and a series of linearly arranged movable photodiodes or photodetectors, or any combination thereof.

[0097] The electric motor 20230 may include a rotatable shaft that operably interfaces with a gear assembly mounted to mesh with a set of drive teeth or a rack on a displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 20225 element corresponds to a number of linear longitudinal translations of the displacement member. The configuration of the gear ring and sensor may be connected to a linear actuator by a rack and pinion configuration or to a rotary actuator by spur gears or other connections. A power supply may supply power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing with a corresponding drive gear of a gear reduction assembly. The displacement member may represent a longitudinally movable emitter member, emitter bar, I-beam, or a combination thereof.

[0098] One rotation of the sensor element associated with the position sensor 20225 can correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance that the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor device can be connected via a gear reduction device in which the position sensor 20225 completes one or more rotations with respect to the full stroke of the displacement member. The position sensor 20225 may complete multiple rotations with respect to the full stroke of the displacement member.

[0099] To provide a unique position signal for two or more rotations of the position sensor 20225, a series of switches (where n is an integer greater than 1) may be used alone or in combination with a gear reduction device. The state of the switch can be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1 + d2 +... dn of the displacement member. The output of the position sensor 20225 is supplied to the microcontroller 20221. The position sensor 20225 of the sensor device may comprise a magnetic sensor, an analog rotary sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0100] The position sensor 20225 may comprise any number of magnetic sensing elements such as, for example, a magnetic sensor classified by measuring the total magnetic field or a vector component of the magnetic field. The technologies used to produce both types of magnetic sensors can encompass many aspects of physics and electronics. Technologies used for sensing magnetic fields include, among others, search coils, fluxgates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetic impedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and microelectromechanical systems-based magnetic sensors.

[0101] The position sensor 20225 of the tracking system 20228 with an absolute positioning system may include a magnetic rotary absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 20225 is connected to the microcontroller 20221 to implement an absolute positioning system. The position sensor 20225 is a low-voltage and low-power component and may include four Hall effect elements in the area of the position sensor 20225 that can be arranged above the magnet. Also, a high-resolution ADC and a smart power management controller may be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the border algorithm, may be provided to implement a concise and efficient algorithm for calculating hyperbolic and trigonometric functions that only require addition, subtraction, bit shifting, and table reference operations. The angular position, alarm bit, and magnetic field information may be transmitted to the microcontroller 20221 via a standard serial communication interface such as a serial peripheral interface (SPI) interface. The position sensor 20225 may provide a resolution of 12 bits or 14 bits. The position sensor 20225 may be an AS5055 chip provided in a small QFN16-pin 4×4×0.85 mm package.

[0102] Tracking system 20228 with an absolute positioning system may include and / or be programmed to implement a feedback controller such as a PID, state feedback, and adaptive controller. The power supply converts a signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by position sensor 20225, other sensor(s) may be provided to measure physical parameters of the physical system. In some aspects, examples of other sensor(s) may include sensor arrangements such as those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM", which is hereby incorporated by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM", which is hereby incorporated by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT", which is hereby incorporated by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include a comparison and combination circuit to combine the calculated response with the measured response using algorithms such as weighted averages and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, inductive resistance, etc. to predict how the state and output of the physical system will be based on knowing the input.

[0103] An absolute positioning system can provide the absolute position of a displacement member upon power-up of the instrument without having to move the displacement member, such as a device actuator, drive bar, knife, etc., backward or forward to reset (zero or home) position, which may require a conventional rotary encoder that simply counts the number of steps taken by the motor 20230 to estimate the position.

[0104] For example, a sensor 20226, such as a strain gauge or a micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the amplitude of the strain exerted on the anvil during a clamping operation, which can indicate, for example, the closing force applied to the anvil. The measured strain can be converted into a digital signal and provided to the processor 20222. Instead of, or in addition to, the sensor 20226, a sensor 20227, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, a sensor 20227, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage a wedge thread, which is configured to cam the staple driver upward to eject a staple and deformably contact the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue when the I-beam is advanced distally by a firing bar. Alternatively, a current sensor 20231 may be used to measure the current consumed by the motor 20230. The force required to advance the firing member can correspond to, for example, the current drawn by the motor 20230. The measured force can be converted into a digital signal and provided to the processor 20222.

[0105] For example, a strain gauge sensor 20226 may be used to measure the force applied to tissue by an end effector. To measure the force exerted by the end effector on the tissue being treated, a strain gauge may be coupled to the end effector. A system for measuring the force applied to tissue grasped by the end effector may include a strain gauge sensor 20226, such as a micro strain gauge, configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 20226 can measure the amplitude or magnitude of the strain exerted on the jaw members of the end effector during a clamping operation, which can indicate tissue compression. The measured strain can be converted to a digital signal and supplied to a processor 20222 of a microcontroller 20221. A load sensor 20227 may measure, for example, the force used to operate a knife element to cut tissue captured between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the captured tissue. The measurements of the magnetic field sensor may also be converted to a digital signal and provided to the processor 20222.

[0106] The measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, respectively measured by sensors 20226, 20227, may be used by the microcontroller 20221 to characterize corresponding values of a selected position of the firing member and / or the velocity of the firing member. In one embodiment, the memory 20223 may store techniques, equations, and / or look-up tables that may be used by the microcontroller 20221 during evaluation.

[0107] The control system 20220 of the surgical instrument or tool may also include a wired or wireless communication circuit for communicating with a surgical hub 20065 as shown in FIG. 4.

[0108] FIG. 6 illustrates an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 that can communicate with a console 20294 or a portable device 20296 through a local area network 20292 and / or a cloud network 20293 via a wired and / or wireless connection. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein for measuring the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in vivo loading unit, that is, a multi-firing loading unit (MFLU), that allows a clinician to fire a plurality of fasteners multiple times without removing the loading unit 20287 from the surgical site to reload the loading unit 20287.

[0109] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multi-firing fastener cartridge that can fire a plurality of fasteners (e.g., staples, clips, etc.) two or more times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener as the fastener is ejected from the multi-firing fastener cartridge.

[0110] The handle 20297 may include a motor coupled to the drive shaft so as to act on the rotation of the drive shaft. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanism or user interface, which may be engaged by a clinician to activate the motor.

[0111] The control interface of the handle 20297 may communicate with the controller 20298 of the handle 20297 to selectively activate the motor and act on the rotation of the drive shaft. The controller 20298 may be disposed within the handle 20297 and may be configured to receive inputs from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 may analyze the inputs from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display visible to the clinician during use of the handle 20297. The display may be configured to display portions of the adapter or loading unit data before, during, or after firing of the instrument 20282.

[0112] The adapter 20285 may include an adapter identification device 20284 disposed therein, and the loading unit 20287 may include a loading unit identification device 20288 disposed therein. The adapter identification device 20284 may communicate with the controller 20298, and the loading unit identification device 20288 may communicate with the controller 20298. It will be understood that the loading unit identification device 20288 may communicate with the adapter identification device 20284 that relays or passes the communication from the loading unit identification device 20288 to the controller 20298.

[0113] Adapter 20285 may also include a plurality of sensors 20286 (one is shown) disposed around it to detect various states of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of firings of the adapter 20285, the peak force of the adapter 20285 during firing, the total amount of force applied to the adapter 20285, the peak recoil force of the adapter 20285, the number of pauses of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide an input to the adapter identification device 20284 in the form of a data signal. The data signals of the plurality of sensors 20286 may be stored within the adapter identification device 20284 or may be used to update the adapter data stored within the adapter identification device 20284. The data signals of the plurality of sensors 20286 may be analog or digital. The plurality of sensors 20286 may include a force gauge for measuring the force exerted on the loading unit 20287 during firing.

[0114] The handle 20297 and the adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally, or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting (e.g., inductively transmitting) energy and signals therebetween. It is also contemplated that the adapter identification device 20284 and the controller 20298 may wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0115] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., the LAN 20292, the cloud 20293, the console 20294, or the portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 20280. For example, the controller 20298 may transmit instrument data including the serial number of a mounting adapter (e.g., adapter 20285) attached to the handle 20297, the serial number of a loading unit (e.g., loading unit 20287) attached to the adapter 20285, and the serial numbers of a plurality of firing fastener cartridges loaded into the loading unit to the console 20294. The console 20294 may then reply to the controller 20298 with data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively. The controller 20298 may display a message on a local instrument display or, alternatively, transmit a message via the transceiver 20283 to the console 20294 or the portable device 20296 to display the message on the display 20295 or the portable device screen, respectively.

[0116] FIG. 7 illustrates a diagram of a situation awareness surgical system 5100 according to at least one aspect of the present disclosure. The data source 5126 can include, for example, a modular device 5102 (which can include sensors configured to detect parameters associated with a patient, an HCP, and the environment, and / or the modular device itself), a database 5122 (e.g., an EMR database including patient records), a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor), an HCP monitoring device 35510, and / or an environmental monitoring device 35512. The surgical hub 5104 can be configured to derive context information regarding a surgical procedure from the data, for example, based on a particular combination of the received data or the particular order in which data is received from the data source 5126. The context information inferred from the received data can include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure that the surgeon is performing, the type of tissue being operated on, or the body cavity that is the subject of the procedure. This function according to some aspects of the surgical hub 5104 for deriving or inferring information regarding a surgical procedure from the received data can be referred to as “situation awareness”. For example, the surgical hub 5104 can incorporate a situation awareness system, which is hardware and / or programming associated with the surgical hub 5104 that derives context information regarding a surgical procedure from received data and / or surgical planning information received from an edge computing system 35514 or an enterprise cloud server 35516.

[0117] The situation recognition system of the surgical hub 5104 can be configured to derive context information from data received from various different data sources 5126. For example, the situation recognition system can include a pattern recognition system or a machine learning system (such as an artificial neural network) trained with training data to correlate various inputs (e.g., data from the database 5122, the patient monitoring device 5124, the modular device 5102, the HCP monitoring device 35510, and / or the environmental monitoring device 35512) with corresponding context information regarding the surgical procedure. The machine learning system can be trained to accurately derive context information regarding the surgical procedure from the provided inputs. In an example, the situation recognition system can include a lookup table that stores pre-characterized context information regarding the surgical procedure in association with one or more inputs (or a range of inputs) corresponding to that context information. In response to a query by one or more inputs, the lookup table can return the corresponding context information of the situation recognition system to control the modular device 5102. In an example, the context information received by the situation recognition system of the surgical hub 5104 can be associated with a specific control adjustment or a series of control adjustments of one or more modular devices 5102. In an example, the situation recognition system can include a further machine learning system, a lookup table, or other such systems that generate or retrieve one or more control adjustments of one or more modular devices 5102 when the context information is provided as an input.

[0118] The surgical hub 5104 incorporating the situation awareness system can provide many advantages to the surgical system 5100. One advantage can include providing improved interpretation of sensed and collected data, which can improve the processing accuracy during the surgical procedure and / or the use of the data. Returning to the previous example, the situation awareness surgical hub 5104 can determine which type of tissue is being operated on, and thus, if an unexpectedly high force is detected to close the end effector of the surgical instrument, the situation awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the tissue type.

[0119] The type of tissue being operated on can affect the adjustments made to the compression speed and load threshold of a surgical stapling and cutting instrument for specific tissue gap measurements. The situation awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic procedure or an abdominal procedure, whereby the surgical hub 5104 can determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (in the case of a thoracic procedure) or stomach tissue (in the case of an abdominal procedure). The surgical hub 5104 can then appropriately adjust the compression speed and load threshold of the surgical stapling and cutting instrument according to the type of tissue.

[0120] The type of body cavity being operated on during a insufflation procedure can affect the function of the smoke evacuation device. The situation awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is using insufflation) and can determine the type of procedure. Generally, since a certain type of procedure can be performed within a specific body cavity, the surgical hub 5104 can appropriately control the motor speed of the smoke evacuation device according to the body cavity being operated on. Thus, the situation awareness surgical hub 5104 can provide a consistent amount of smoke evacuation for both thoracic and abdominal procedures.

[0121] The type of procedure being performed can affect the energy level optimal for the operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopic procedures, the end effector of an ultrasonic surgical instrument or an RF electrosurgical instrument is immersed in fluid and may require a higher energy level. The Situational Awareness Surgical Hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The Surgical Hub 5104 can then adjust the RF power level of the generator or the ultrasonic amplitude (e.g., "energy level") to compensate for the fluid-filled environment. In connection therewith, the type of tissue being operated on can affect the energy level optimal for the operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The Situational Awareness Surgical Hub 5104 can determine which type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument, respectively, according to the tissue shape expected for the surgical procedure. Further, the Situational Awareness Surgical Hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument not only for each procedure but also over the course of the surgical procedure. The Situational Awareness Surgical Hub 5104 can determine which step of the surgical procedure is being performed or will be performed next and then update the control algorithm of the generator and / or the ultrasonic surgical instrument or the RF electrosurgical instrument to set the energy level to an appropriate value for the type of tissue expected according to the step of the surgical procedure.

[0122] In an example, the surgical hub 5104 can derive data from an additional data source 5126 to improve conclusions drawn from one data source 5126. The situation awareness surgical hub 5104 can enhance data received from the modular device 5102 with context information constructed regarding the surgical procedure from other data sources 5126. For example, the situation awareness surgical hub 5104 can be configured to determine whether hemostasis has occurred (e.g., whether bleeding has stopped at the surgical site) according to video or image data received from a medical imaging device. The surgical hub 5104 can be further configured to make a determination regarding the integrity of a staple line or tissue weld by comparing physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device communicatively coupled to the surgical hub 5104). The situation awareness system of the surgical hub 5104 can provide additional context when analyzing visualization data in consideration of physiological measurement data. The additional context can be useful when the visualization data may not be conclusive or may be incomplete by itself.

[0123] For example, if it is determined that the use of an instrument is required in a subsequent step of the procedure, the situation awareness surgical hub 5104 can actively activate the generator to which an RF electrosurgical instrument is connected. Actively activating the energy source can enable the instrument to be ready for use as soon as the preceding step of the procedure is completed.

[0124] The situation awareness surgical hub 5104 can determine whether the current step or subsequent steps of a surgical procedure require different views or magnifications on a display according to the features of the surgical site that a surgeon is expected to view. The surgical hub 5104 can actively change the displayed view (e.g., supplied from a medical imaging device for a visualization system) as appropriate, such that the display automatically adjusts throughout the surgical procedure.

[0125] The situation awareness surgical hub 5104 can determine which step of a surgical procedure is being performed or will be performed next and whether specific data or a comparison between data is required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up a data screen based on the step of the surgical procedure being performed without waiting for a surgeon to request specific information.

[0126] During the setup of a surgical procedure or during the surgical procedure itself, errors can be checked. For example, the Situational Awareness Surgical Hub 5104 can determine whether the operating room is appropriately or optimally set up for the surgical procedure to be performed. The Surgical Hub 5104 can determine the type of surgical procedure being performed, read the corresponding checklist, product locations, or setup requirements (e.g., from memory), and then be configured to compare the current operating room layout to the standard layout for the type of surgical procedure that the Surgical Hub 5104 has determined is being performed. In some examples, the Surgical Hub 5104 can compare a list of items for the procedure and / or a list of devices paired with the Surgical Hub 5104 to the recommended or expected manifest of items and / or devices for a given surgical procedure. If there is any discrepancy between the lists, the Surgical Hub 5104 can provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, HCP monitoring device 35510, environmental monitoring device 35512, and / or other surgical supplies are missing. In some examples, the Surgical Hub 5104 can determine, for example, the relative distance or relative position of the modular device 5102 and the patient monitoring device 5124 via a proximity sensor. The Surgical Hub 5104 can compare the relative position of the devices to the layout recommended or expected for a particular surgical procedure. If there is any discrepancy between the layouts, the Surgical Hub 5104 can be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0127] The situation awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making a mistake or deviating from a series of actions expected during a surgical procedure. For example, the surgical hub 5104 can determine the type of surgical procedure being performed, read a corresponding list of steps or order of device use (e.g., from memory), and then compare the steps being taken or devices being used during the surgical procedure to the steps or devices expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. The surgical hub 5104 can provide an alert indicating that an unexpected action is being taken at a particular step in the surgical procedure or that an unexpected device is being utilized.

[0128] Surgical instruments (and other modular devices 5102) can be adjusted to suit the specific context of each surgical procedure (such as adjustment to different tissue types) and can verify actions during the surgical procedure. The next steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the procedure.

[0129] Surgical autonomous systems, devices, and methods can include aspects of integration with other medical devices, data sources, processes, and institutions. Surgical autonomous systems, devices, and methods can include, for example, aspects of integration with a computer-implemented bidirectional surgical system and / or one or more elements of a computer-implemented bidirectional surgical system. Surgical systems, surgical autonomous systems, and autonomous surgical systems can be compatible as described herein.

[0130] Referring to FIG. 8, an overview of the surgical autonomous system 47000 can be provided. The surgical instrument A 47005 and / or the surgical instrument B 47010 can be used in a surgical operation as part of the surgical system 47000. The surgical hub 47040 can also be configured to regulate the information flow to the display of the surgical instrument. For example, the surgical hub may be described in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. Exemplary surgical instruments suitable for use with the surgical system 47000 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0131] FIG. 8 shows an example of the surgical autonomous system 47000. The system 47000 can be used to perform a surgical operation on a patient lying on an operating table in an operating room. The robotic system can be used as part of the surgical system in a surgical operation. For example, the robotic system may be described in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. The robotic hub can be used to process an image of the surgical site and then display it to the surgeon through the surgeon's console.

[0132] Other types of robotic systems can be readily adapted for use with the surgical system 47000. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019 / 0201137 (A1) (U.S. Patent Application No. 16 / 209,407) entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0133] Various examples of cloud-based analysis methods implemented by the cloud and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569 (A1) (U.S. Patent Application No. 16 / 209,403) entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB", filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0134] In various aspects, the imaging device may be used in the surgical system 47000 and may include at least one image sensor and one or more optical components. Suitable image sensors can include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0135] The optical components of the imaging device may include one or more light sources and / or one or more lenses. The one or more light sources may be directed to illuminate a portion of the surgical field. The one or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0136] One or more light sources may be configured to irradiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum, which may also be referred to as the optical spectrum or emission spectrum in some cases, is a portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by the human eye) and may be referred to as visible light or simply light. A typical human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0137] The invisible spectrum (e.g., non-emission spectrum) is a portion of the electromagnetic spectrum that is located below and above the visible spectrum (i.e., wavelengths less than approximately 380 nm and greater than approximately 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet, x-rays, and gamma ray electromagnetic radiation.

[0138] In various aspects, the imaging device may be configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-ureteroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0139] The imaging device may employ multispectral monitoring to distinguish topography from the underlying structure. A multispectral image captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelength can be separated by a filter or by using an instrument having sensitivity to a specific wavelength including frequencies beyond the visible light range, e.g., IR, and light from ultraviolet. Spectral imaging enables extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading “Advanced Imaging Acquisition Module” in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after a surgical task for performing one or more of the above-described tests on the treated tissue has been completed. It is understood that strict sterilization of the operating room and surgical instruments is required during any surgical procedure. The strict hygiene and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize anything that comes into contact with the patient or enters the sterile field, including the imaging device and its accessories and components. It will be understood that the sterile field can be considered a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding the surgically prepared patient. The sterile field can include properly attired and scrubbed team members, as well as all equipment and fixtures within that area.

[0140] As described with respect to FIGS. 2, 3, 5, and 7, surgical instrument A 47005 and / or surgical instrument B 47010 can include one or more capabilities (e.g., capabilities 47015 associated with surgical instrument A 47005, which can include B, Z, and D, and capabilities 47020 associated with surgical instrument B 47010, which can include C, F, and E). The capabilities can be associated with functions that the surgical instrument can perform. Examples of functions can be described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed Dec. 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. For example, if the surgical instrument is an end cutter, one of the capabilities can be to excise tissue (e.g., the tissue surrounding the colon when a surgeon performs a colectomy). The capabilities can include surgical tasks as described with respect to FIG. 9. For example, the ability to excise tissue can include control of an energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, reload alignment slot management, and / or the like.

[0141] The capabilities (e.g., each of the capabilities) of surgical instrument A 47005 and / or surgical instrument B 47010 can be associated with respective autonomy levels (e.g., the capability 47015 associated with surgical instrument A 47005 can be associated with autonomy level 47030, and the capability 47020 associated with surgical instrument B 47010 can be associated with autonomy level 47025). For example, the autonomy levels may be classified by numbers such as 1, 2, 3. 1 may represent the autonomy level with the least manual input when compared to other autonomy levels. For example, the autonomy level with the least manual input may result in the user of the surgical instrument having to perform fewer surgical tasks associated with the capability when compared to other autonomy levels. If the surgical instrument is an end cutter and the capability is to excise tissue, autonomy level 1 may result in the surgeon only having to perform control of the energy source, while autonomy level 2 may result in the surgeon having to perform control of the energy source, body orientation, and / or body position. Tasks not performed by the surgeon can be performed using autonomous functions.

[0142] Data can be generated based on the performance of surgical instrument A 47005 and / or B 47010 (e.g., by a monitoring module located in surgical hub 47040 or locally by the surgical instrument as described with respect to FIG. 11). The data can be related to how the current autonomy level at which the surgical instrument is operating is performing with respect to performance. For example, the data may be associated with body measurements, physiological measurements, and / or the like as described with respect to FIGS. 10 and 11. The measurements are described in more detail under the title "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements" in U.S. Patent Application No. 17 / 156,28, filed Nov. 10, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0143] The data instruction 47035 may be sent, for example, to a surgical hub 47040 where it can be evaluated. In an example, the instruction 47035 may be sent to a cloud service (e.g., Amazon Web Services) as described herein. The data may be used as an input in an analysis module, for example, to check whether the performance of a surgical instrument is within a satisfactory range. The surgical instrument may be provided with capabilities associated with a current autonomy level (e.g., autonomy level 1). If data related to performance is out of range (e.g., exceeds a threshold as described with respect to FIGS. 11 and 9), the surgical hub 47040 may send back the instruction 47035 to the surgical instrument (e.g., surgical instrument A 47005 and / or surgical instrument B 47010) to switch the level of autonomy associated with the capabilities. For example, tissue resection may be switched from autonomy level 1 to autonomy level 2 if data related to performance exceeds a threshold.

[0144] FIG. 9 shows an example of an autonomy module 47050 associated with a surgical instrument 47045 and a surgical instrument capability 47060. The autonomy module 47050 may be associated with the surgical instrument 47045. In an example, the autonomy module 47045 may be part of the software of the surgical instrument. In an example, the autonomy module 47045 may be part of the surgical hub as described with respect to FIG. 8. The autonomy module 47045 may continue to track the link between the surgical instrument capability 47060 and the autonomy level. For example, the autonomy module 47050 may refer to a database (e.g., a Structured Query Language (SQL) database) that maintains the link between the capability 47060 and the autonomy level at a given time (e.g., maintains that at the current time, capability A is operating at autonomy level 1), for example, via a query. The database may be updated based on the autonomy level associated with the change in capability. In an example, the database may link one or more tasks 47055 to the autonomy level. In such a case, the capability 47060, the autonomy level associated with the capability 47060, and the surgical task 47055 associated with the autonomy level may be linked and queried by the autonomy module.

[0145] In an example, the autonomy module 47050 may include all surgical tasks associated with the surgical instrument 47045. The autonomy module 47050 can add surgical tasks to a data structure (e.g., a list) associated with each respective ability 47060. The module 47050 may specify one or more surgical tasks 47055 for the surgical instrument 47045 to autonomously perform, based on the ability 47060 and the autonomy level, as described herein. For example, the surgical instrument 47045 may be an end cutter. The ability of the end cutter may be to excise tissue, and the autonomy level of this ability 47060 may be 1. The autonomy module 47050 may specify that control of the energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, and reload alignment slot management are to be autonomously performed. In such a case, when the ability 47060 switches to an autonomy level 2, the autonomy module 47050 can specify that fewer tasks 47055 are to be autonomously performed. For example, in the case of autonomy level 2, the autonomy module 47055 may specify body position, anvil jaw force, and reload alignment slot management to be autonomously performed. The other tasks 47055 may be manually performed by the surgeon.

[0146] The surgical instrument 47045 may be provided with an ability 47060, and each ability 47060 can operate only at a particular autonomy level. In the example, the autonomy module 47050 can check with a rules engine whether an ability 47060 is permitted to operate at a given autonomy level. For example, ability A may be organ mobilization. Organ mobilization may enable the surgical instrument 47045 to operate only at autonomy level 1 or autonomy level 3 (e.g., as shown in FIG. 9). Ability B may be anastomosis. Anastomosis may enable the surgical instrument 47045 to operate only at autonomy level 2 (e.g., as shown in FIG. 9).

[0147] FIG. 10 shows an example of a computer-implemented autonomous surgical system. The system may include a processor associated with a surgical instrument 47065 and a processor associated with a surgical hub 47115. The surgical instrument processor can be connected via a system bus to communication (e.g., a communication module), storage 47080, memory (e.g., non-volatile memory), a management module, an actuator, and a sensor, an input interface (e.g., which can obtain (47075) measurement values 47070 such as physical, physiological, and vision-based from an external source as described with reference to FIGS. 9 and 11), and an output interface. The system bus can be a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures including, but not limited to, a 9-bit bus, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MSA), an Extended ISA (EISA), an Intelligent Drive Electronics (IDE), a VESA Local Bus (VLB), a Peripheral Component Interconnect (PCI), a USB, an Advanced Graphics Port (AGP), a Personal Computer Memory Card International Association bus (PCMCIA), a Small Computer System Interface (SCSI), or any other proprietary bus.

[0148] The surgical hub processor can be connected via the system bus to communication, storage 47080, memory (e.g., non-volatile memory), an input / output interface, an analysis module, and a management module.

[0149] Each processor may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB of single-cycle flash memory or other non-volatile memory with a maximum of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle serial random access memory (SRAM), an internal read-only memory (ROM) with StellarisWare® software, 2KB of electrically erasable programmable read-only memory (EEPROM) and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, the details of which are available in the product datasheet.

[0150] In an example, each processor may include a safety controller with two controller-based families such as TMS570 and RM4x, also known by the trade name Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, while providing scalable performance, connectivity, and memory options, and providing a high level of integrated safety mechanisms.

[0151] Examples of system memory include volatile memory and non-volatile memory. The basic input / output system (BIOS), which contains basic routines for transferring information between elements within a computer system during startup and the like, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. An example of volatile memory is random access memory (RAM), which functions as an external cache memory. Further, RAM is available in many forms such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM).

[0152] The system may include removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage include, but are not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards or memory sticks. Additionally, disk storage can include the above storage media, either independently or in combination with other storage media. Other storage media include, but are not limited to, optical disk drives such as compact disc ROM devices (compact disc ROM, CD-ROM), compact disc recordable drives (compact disc recordable drive, CD-R drive), compact disc rewritable drives (compact disc rewritable drive, CD-RW drive) or digital versatile disc ROM drives (digital versatile disc ROM drive, DVD-ROM). A removable or non-removable interface may be used to facilitate connection of the disk storage device to the system bus.

[0153] It should be understood that the system may include software that functions as a mediator between the described user and the basic computer resources in a suitable operating environment. Examples of such software include operating systems. In an example, the operating system may be associated with management modules for the surgical instrument 47065 and the surgical hub 47115, respectively. The operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may utilize the resource management by the operating system via program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0154] A user can input commands or information into the system via an input device connected to the I / O interface. These and other input devices are connected to the processor via an interface port and through the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USBs. Output devices use some of the same types of ports as input devices. Thus, for example, a USB port can be used to provide input to and output information from a computer system to an output device. The output device may be a surgical hub 47115. The output device may be a surgical instrument 47065. Output adapters may be provided to illustrate, among other output devices that may require special adapters, that several output devices such as monitors, displays, speakers, and printers can exist. Output adapters can include video and sound cards that provide connection means between the output device and the system bus, but this is by way of example and not limiting. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0155] The system can operate in a network environment using a logical connection to one or more remote computers, such as a cloud computer (e.g., as described with respect to FIG. 8) or a local computer. The remote cloud computer can be, for example, a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, or other common network nodes, but typically includes many or all of the elements described with respect to a computer system. For simplicity, only a memory storage device is illustrated along with the remote computer. The remote computer can be logically connected to the computer system via a network interface and subsequently physically connected via a communication connection. The network interface can include communication networks such as local area networks (LANs) and wide area networks (WANs). Examples of LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, and the like. Examples of WAN technologies include circuit-switched networks such as point-to-point links, Integrated Services Digital Network (ISDN) and its variants, packet-switched networks, and Digital Subscriber Line (DSL), but are not limited thereto.

[0156] In various aspects, a computer system may include an image processor, an image processing engine, a media processor, or any special digital signal processor (DSP) used for processing digital images. The image processor can enhance speed and efficiency using parallel computing with single instruction, multiple data (SIMD), or multiple instruction, multiple data (MIMD) techniques. The digital image processing engine can perform various tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0157] The communication connection part may refer to the hardware / software used to connect a network interface to a bus. For the sake of exemplary clarity, the communication connection part is shown inside the computer system, but the communication connection part may be outside the computer system. For illustrative purposes only, the hardware / software required for connection to a network interface can include internal and external technologies such as ordinary telephone-grade modems, cable modems, and DSL modems including modems, ISDN adapters, and Ethernet cards.

[0158] The surgical instrument 47065 may include one or more hardware components such as an actuator, a sensor, etc. When the surgical instrument is performing a surgical operation, data can be generated from the hardware components, communicated to the memory of the surgical instrument, and archived in the storage 47080 of the surgical instrument. The data may be related to the performance of the surgical instrument, the physical state, the physiological state, the vision-based state, etc. As described herein, the data can be received by the surgical instrument from an external source (47075). The physical state may be related to the force applied by the surgical instrument (e.g., mechanical grasping, etc.) and / or the kinematics of the surgical instrument 47065 (e.g., the position, speed, and / or orientation of the instrument).

[0159] Data (e.g., an instruction 47150 of data) may be transmitted via communication to a surgical hub 47115 as described herein. The surgical hub 47115 may use the data as an input in an analysis module. The analysis module may evaluate whether the data exceeds a threshold (e.g., within a range of satisfactory results as described with respect to FIGS. 9 and 11). In an example, the analysis module may be a cloud service such as a Lambda function of Amazon Web Services (trademark) (AWS). The surgical hub 47115 may archive the data in storage 47120 (e.g., the data may be used for a subsequent surgery). The data may be analyzed (e.g., by the analysis module), and an instruction 47145 of the analyzed data may be sent back to the surgical instrument 47065 as an input.

[0160] FIG. 11 shows an example of monitoring the execution of a surgical task associated with an autonomy level. The monitoring module 47155 may be part of a surgical instrument, a surgical hub, or a remote device (e.g., a third-party cloud service). In an example, aspects of the monitoring module 48155 may be distributed among multiple devices (e.g., each device may be responsible for executing a particular set of instructions). The monitoring module 47155 may be the analysis module described with respect to FIG. 10. The monitoring module 47155 may evaluate whether the performance of the surgical instrument capabilities operating at the autonomy level is within an acceptable (e.g., satisfactory) range.

[0161] The monitoring module 48155 can obtain data associated with the surgical instrument and / or the patient 47175, as shown in FIG. 11. The data may be generated from an external source (e.g., something wearable by the patient 47175 that measures biomarker data), and an indication of the data may be sent to the monitoring module 47155 via a message. The message may be in response to the monitoring module 47155 requesting the data via a request message. The data may be in raw form or may be converted by a processor (e.g., a surgical hub processor or a surgical instrument processor as described with respect to FIG. 10) into a form suitable for analysis.

[0162] The data may be related to the performance of the surgery. For example, the data may be related to the measured physiological states of the patient 47175, the surgeon, and / or the staff participating in the surgery. For example, the patient 47175 may be wearing a heart rate monitoring wearable. The monitoring wearable is described in more detail under the title "Method Of Adjusting A Surgical Parameter Based On Biomarker Measurements" in U.S. Patent Application No. 17 / 156,28, filed on November 10, 2021, the disclosure of which is incorporated herein by reference in its entirety. In such a case, data related to the patient's heart rate may be obtained and used by the monitoring module. The surgeon may be wearing a headband wearable that measures stress via a sweat sensor (e.g., by estimating the level of cortisol being generated by the surgeon). The data related to the measured physiological state may be compiled in the measured physiological module 47170, which may be part of the monitoring module 47155.

[0163] Data may be related to the measured physical states of the instrument, patient 47175, surgeon, and / or staff. For example, a potentiometer and / or sensor may be disposed on the surgical instrument, and data related to the movement and / or orientation of the instrument may be generated and transmitted to the monitoring module 47155. For example, an instrument that autonomously performs tasks based on the autonomy level described with respect to FIG. 9 can move from position A to position B. The speed at which the instrument has moved can be calculated (e.g., based on the calculation of readings from a potentiometer), and an indication of the speed can be transmitted to the monitoring module. In an example, the measured physical state may be obtained via data generated by a vision-based device. This data may be compiled in the measured physical module 47160. In one example, the physical and / or physiological state may be the subject of the operation. In one example, a state other than the physical and / or physiological state may be the subject of the operation.

[0164] The monitoring module 47155 can communicate with the surgical instrument. The surgical instrument may be equipped with capabilities, and the surgical instrument may operate at a current autonomy level, e.g., autonomy level 1 47190 as shown in FIG. 11, to perform the capabilities. Operating at autonomy level 1 47190 may result in the surgical instrument autonomously performing some tasks associated with the capabilities. For example, the automated tasks 47210 may be clamping, positioning, and orientation, as shown in FIG. 11. This may leave other tasks associated with the firing and stapling capabilities that are to be performed manually (e.g., by the surgeon). Data related to the automated tasks 47210 and the manual tasks 47215 can be generated and transmitted to the monitoring module 47155. The data may be compiled in the measured physical module 47160 as described herein.

[0165] The monitoring module 47155 may include an ideal physical module 47180 and / or an ideal physiological module 47185. The module may include ideal values associated with several physical and physiological states. The ideal values may be dynamic and may change based on the surgical context. For example, the ideal value associated with the surgeon's heart rate may increase if the surgeon is under the stress of the operation. The ideal values can be generated based on past data as described with respect to FIG. 9. The ideal values may be manually entered, for example, by the surgeon or other medical professionals. In the example, the measured physical module 47160, the measured physiological module 47170, the ideal physical module 47180, and the ideal physiological module 47185 may be organized in different formats.

[0166] The monitoring module 47155 can compare the measured physical module 47160 and the measured physiological module 47170 with the ideal physical module 47180 and the ideal physiological module 47185 respectively. In the example, the analysis module may be included in the monitoring module 47155 and may be responsible for evaluating the comparison. The difference between the measured module and the ideal module can be calculated and a delta output 47200 can be generated. In the example, weights can be assigned to each physiological state and the weights can be considered when generating the delta output 47200. In the example, only a subset of the measured physical state and / or the measured physiological state can be compared with the ideal state that can be determined based on the surgical context as described with respect to FIG. 9.

[0167] The delta output 47200 can be compared with a trigger (e.g., a trigger event), for example, to evaluate whether the trigger is satisfied. This trigger may represent whether the difference between the measured module and the ideal module exceeds an acceptable value and / or range. In the example, the delta output 47200 may be generated for each physiological and / or physical state, and each delta output 47200 may be compared with its respective trigger.

[0168] Based on the trigger being satisfied, the level of autonomy associated with the surgical instrument capabilities may switch to another level of autonomy (47220). In an example, the level of autonomy may switch only to an allowed level of autonomy for the capabilities, as described with respect to FIG. 9. As shown in FIG. 11, the trigger was satisfied and the level of autonomy at which the surgical instrument was operating to perform its capabilities was switched from level of autonomy 1 47190 to level of autonomy 2 47230 (47220). Level of autonomy 2 47230 may result in fewer tasks being performed autonomously (e.g., automated tasks 47235) and more tasks being performed manually (e.g., manual tasks 47240). For example, as shown in FIG. 11, when the surgical instrument is at level of autonomy 2 47230, it may perform the clamp autonomously and may result in the surgeon performing the firing, stapling, positioning, and orientation manually. The update of the level of autonomy can be sent to the database as described with respect to FIG. 9.

[0169] FIG. 12 shows an example of the relationship 47245 between the error magnitude and the level of autonomy. In an example, the error magnitude may be associated (e.g., linked) with each level of autonomy. The error magnitude may be used as the threshold described with respect to FIG. 11. For example, the monitoring module described with respect to FIG. 11 may generate an output (e.g., a delta output) based on the difference between the measured condition and the ideal condition. The output may be compared to the error magnitude (e.g., used as a threshold) to determine whether the current level of autonomy is resulting in acceptable results. For example, as shown in FIG. 12, level of autonomy 1 may be associated with an error magnitude of 0.01%. If the output from the monitoring module (e.g., which may be a numerical value) exceeds 0.01%, an instruction may be sent to the surgical instrument to switch the level of autonomy, for example, to level of autonomy 2.

[0170] As shown in FIG. 12, the magnitude of the error may increase as the level of autonomy increases, and the increased level of autonomy results in fewer tasks being performed autonomously, as described with respect to FIG. 9. When the level of autonomy is switched, an instruction regarding the switch may be sent to the monitoring module, and the instruction may include the updated magnitude of the error. For example, when the level of autonomy is switched to autonomy level 2, an error magnitude of 0.1% may be sent to the monitoring module. The monitoring module may include a database, such as the databases described with respect to FIGS. 9 and 11. The monitoring module can send a message to the database to update both the current level of autonomy at which the surgical instrument is operating and the value of the magnitude of the error associated with the updated level of autonomy. The monitoring module can infer the level of autonomy based on the magnitude of the error.

[0171] The magnitude of the error associated with the level of autonomy can be determined. For example, the surgical hub may determine the magnitude of the error of the level of autonomy based on past data. The past data can indicate a correlation between the likelihood of an adverse outcome and the magnitude of the error. The past data may be based on an analysis of what occurred when the autonomous function (e.g., associated with the level of autonomy) was used to perform an ability in the past. The surgical hub may include an error magnitude module that calculates the magnitude of the error that minimizes the likelihood of an adverse outcome based on past data. In an example, the error magnitude module may determine the magnitude of the error based on weighting the benefits of performing the ability using the level of autonomy versus performing the ability manually. For example, performing the ability manually may introduce a likelihood of an adverse outcome that the error magnitude module may consider when determining the magnitude of the error. As described with respect to FIG. 11, an indication of the magnitude of the error can be sent to the monitoring module.

[0172] The magnitude of the error associated with each level of autonomy can be dynamic. For example, a surgical hub can send to a monitoring module a range of acceptable error magnitudes for a level of autonomy. When a surgical instrument is performing an ability, surgical context data can be sent to the monitoring module from, for example, a surgical hub, a surgical instrument, or other devices associated with the surgery (e.g., a wearable that a surgeon may wear). The monitoring module can send an instruction to adjust the magnitude of the error based on the surgical context data. For example, the surgical context data can indicate that an important step of the surgery is being performed. The monitoring module can obtain the data and send an instruction to reduce the magnitude of the error (e.g., the reduction results in exceeding a threshold in a state where fewer errors occur).

[0173] Figure 13 shows an example of the relationship between the magnitude of the error and the level of autonomy.

[0174] As shown in FIG. 13, a high level of autonomy 47250 (e.g., resulting from a large number of tasks associated with surgical instrument capabilities being autonomously performed) can be associated with a low magnitude of error (e.g., the total magnitude of the error). In the example, the high level of autonomy 47250 can be represented as level of autonomy 1 as described herein. A low level of autonomy 47260 (e.g., resulting in a small number of tasks associated with surgically instrument capabilities being autonomously performed) can be associated with a high magnitude of error (e.g., the total magnitude of the error). As described with respect to FIG. 12, the magnitude of the error can be related to a threshold used to determine whether the level of autonomy at which the surgical instrument is operating should be switched.

[0175] As described with respect to FIGS. 12 and 9, the error magnitude can be determined based on past data, surgical context (which may include environmental data such as the number of staff members in the operating room (OR), surgical instrument data as described with respect to FIG. 11, etc.). The error magnitude can be determined based on the surgical tasks being autonomously performed. For example, the ability of an end cutter may be to excise tissue. This may be set to autonomy level 1 to perform this ability, which may result in the autonomous execution of surgical tasks such as controlling the energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, and reload alignment slot management (for example, these are tasks associated with excising tissue). Data related to one or more of these surgical tasks, such as knob orientation, may be acquired by the surgical hub, and the surgical hub may use this data when determining the error magnitude. For example, the data can indicate that the knob orientation is a task that is likely to be successfully performed when autonomously executed. In such a case, the surgical hub may determine a low error magnitude. In the example, the error magnitude may be sent to a monitoring module as described with respect to FIG. 12, and the monitoring module may infer the autonomy level being used by the instrument. The data can indicate that stapling is a task that is less likely to be successfully performed when autonomously executed. In such a case, the surgical hub may determine a high error magnitude. Data related to multiple tasks can be used, for example, by the surgical hub to determine the error magnitude. For example, both data from knob orientation and data from stapling can be sent to the surgical hub. The surgical hub can weight both data sets and determine an intermediate level of autonomy 47255 to be used, for example, because the knob orientation is a task that is likely to be successfully performed when autonomously executed and stapling is a task that is less likely to be successfully performed when autonomously executed.The magnitude of an error can be determined for each task associated with the ability being executed. For example, a surgical hub can be used to determine the magnitude of a knob orientation error, which can be used when evaluating whether the level of autonomy associated with the knob orientation should be switched, as described with respect to FIG. 12, and the magnitude of a stapling error, which can be used when evaluating whether the level of autonomy associated with stapling should be switched.

[0176] FIG. 14 shows an example of the relationship between the level of autonomy, machine learning, and surgical tasks. The level of autonomy and the level of automation may be used interchangeably herein.

[0177] As described herein, autonomously performing a surgical task associated with a surgical instrument ability may include using a machine learning framework. The machine learning framework is described in more detail under the title "Method for Surgical Simulation" of U.S. Patent Application No. 17 / 332,593, filed May 27, 2021, the disclosure of which is incorporated herein by reference in its entirety. For example, an input (e.g., a parameter) associated with a surgical task may be sent to a machine learning model 47275, and the machine learning model may be trained to autonomously perform the surgical task 47285 (47280). The machine learning model 47275 may be involved in determining the level of autonomy 47265 (e.g., level of autonomy 1) at which the surgical task 47285 should be performed. Data associated with the level of autonomy 47265 may be generated and sent to the machine learning model 47275 in the form of feedback 47270, and the model 47275 may adjust the input(s) associated with the surgical task 47285 and / or change the level of autonomy 47265 at which the surgical instrument 47285 operates. The machine learning model 47275 may be disposed locally (e.g., as a module) on the surgical instrument or on a remote device.

[0178] The dynamic variables associated with autonomously performing surgical task 47285 can be updated by a machine learning algorithm based on performance metrics, as described with respect to FIG. 11. For example, the autonomy level 47265 associated with surgical task 47285 can be updated based on the magnitude of an error exceeded (e.g., a threshold), as described with respect to FIG. 12. The machine learning model 47275 can use real-world data sets from previous surgeries when determining whether the magnitude of an error has been exceeded and / or when determining the autonomy level 47265 associated with surgical task 47285. The real-world data sets from previous surgeries can be an aggregation of procedures performed by surgeons, surgeries from that facility, and / or an aggregation of surgeons using hubs within the same network.

[0179] The machine learning algorithm can be updated by data from cloud and / or remote systems that can compile, for example, best practices, regional data regarding surgeries, and / or results and usage steps from any number of other facilities around the world.

[0180] Real-world information can be derived, for example, from surgical outcomes from a region, population, etc., and / or can be interpolations and / or aggregations of sub-biomarker measurements and results.

[0181] For example, the machine learning model 47275 based on a GAN (GAN model) can be trained using past surgical data. The GAN model can model the data patterns by which a list of surgical tasks 47285 can be autonomously performed given a surgical step. The GAN model can model the probability distribution of surgical tasks 47285 that exist in past surgical data. That is, when the GAN model generates a surgical task 47285 from a list of possible surgical tasks 47285, the surgical task 47285 is generated with a probability according to the probability distribution.

[0182] Machine learning can be part of a technology platform called cognitive computing (CC), which can constitute various fields such as computer science and cognitive science. A CC system can learn at scale, reason purposefully, and interact with humans and nature. A CC system can be capable of autonomously performing surgical tasks.

[0183] The output of the machine learning training 47280 process may be a model 47275 for predicting results for a new dataset. For example, a linear regression learning algorithm can be a cost function that minimizes the prediction error of a linear prediction function during the training process by adjusting the coefficients and constants of the linear prediction function. When a minimum value can be reached, the linear prediction function with the adjusted coefficients can be considered trained and may constitute the model 47275 generated by the training 47280 process. For example, a neural network (NN) algorithm for classification (e.g., a multilayer perceptron (MLP)) can include a hypothesis function represented by a network of layers of nodes interconnected by weighted connections with biases assigned. The hypothesis function can be a non-linear function (e.g., a highly non-linear function) that includes a linear function and a nested logistic function with an outermost layer consisting of one or more logistic functions. The NN algorithm can include a cost function for minimizing classification errors by adjusting the biases and weights through the processes of forward propagation and backpropagation. When a global minimum value can be reached, the optimized hypothesis function with the adjusted layers of biases and weights can be considered trained and may constitute the model 47275 generated by the training process 47280.

[0184] FIG. 15 shows an example of determining an autonomy level at which a surgical instrument performs a surgical task. As described with respect to FIG. 9, the surgical task(s) can be associated with the capabilities of the surgical instrument, e.g., tissue resection. The surgical steps can be associated with the capabilities.

[0185] The surgery can be in accordance with a surgical plan 47290 that outlines the surgical steps to be performed (e.g., surgical step 1 47295, surgical step 2 47300, surgical step 3 47305, surgical step 4 47310, and surgical step K 47315). The surgical steps can be performed sequentially. In the example, the surgical steps can be performed in parallel. A surgical hub or other device can obtain the name of the surgery to be performed and generate a surgical plan based on past data related to the success of previous surgeries or previous simulations of the surgery (e.g., performed at the facility). The surgical hub can consider data (e.g., additional data) when generating a surgical plan, such as the surgeon's level of experience.

[0186] One or more surgical steps (e.g., each surgical step) can include the surgical task(s) to be performed by the surgical instrument. As described with respect to FIG. 11, one or more of the tasks can be performed autonomously, and one or more can be performed manually, which can be determined based on an autonomy level associated with the capabilities (e.g., surgical steps). In the example, for instance, if the capabilities are associated with the highest autonomy level (e.g., the "fully autonomous" level as shown in FIG. 12), all surgical tasks can be performed autonomously.

[0187] As described with respect to FIG. 14, a machine learning model may be used to determine the autonomy level of a surgical step (e.g., machine learning model 47320 associated with surgical step 147295 and machine learning model 47340 associated with surgical step 4). In an example, the parameters of the machine learning model may be input by user 47355. Machine learning model 47320 may output, for surgical step 147295, that tasks 147360 to N47375 should be executed autonomously and tasks N+147365 to Z47380 should be executed manually (e.g., by a surgeon or another surgical staff member). The machine learning framework can receive feedback regarding the execution of a surgical step. When the performance exceeds a threshold, the machine learning model may obtain a message that the autonomy level should be switched. In such a case, the model may determine an updated autonomy level based on feedback regarding the execution of a surgical step that may include one or more of real-time surgical data, user data, surgical environment data, surgical instrument data, task data, or past data. The update may be performed during a surgical step. The update may be performed based on a transition from one surgical step to another (e.g., during the transition from surgical step 1 to surgical step 2). The updated autonomy level may be associated with a different set of surgical tasks to be automated (e.g., executed autonomously). For example, based on the transition to surgical step 447310, tasks 147360 to X47400 may be executed autonomously and tasks X+147395 to L47405 may be executed manually. A message 47410 (e.g., a simple notification service (SNS)) may be sent to the surgeon or other surgical staff member of the updated autonomy level. The surgical tasks to be executed autonomously and manually may be sent via the message.

[0188] FIG. 16 shows an exemplary flowchart 47415 for automating surgical tasks associated with an autonomy level. At 47420, the device may receive instructions for surgical tasks to be performed using surgical instruments. The capabilities of the surgical instruments may be associated with a level of automation.

[0189] At 47425, the device may monitor the execution of the surgical task while the surgical instrument is operating at a first level of automation associated with the level of automation. The capabilities of the surgical instruments may include a set of surgical instrument tasks, and the level of automation may be associated with automating one or more surgical instrument tasks from the set of surgical instrument tasks.

[0190] At 47430, the device may detect a trigger event associated with the execution of the surgical task and, for example, based on the trigger event, switch the operation of the surgical instrument from the first level of automation to a second level of automation associated with a plurality of levels of automation. The execution of the surgical task can be based on real-time surgical data. The real-time surgical data may include one or more of user data, surgical environment data, surgical instrument data, task data, past data, etc.

[0191] At 47435, the device may detect the trigger event by comparing the execution of the surgical task with a trigger event threshold and, based on the trigger event, switch the operation of the surgical instrument from the first level of automation to the second level of automation. The second level of automation may be associated with automating fewer surgical instrument tasks when compared to the first level of automation. Monitoring the execution of the surgical task may include comparing one or more of the real-time surgical data with their respective ideal surgical data.

[0192] In an example, the device may operate at a first level of automation among levels of automation associated with a surgical task. The device may obtain an instruction to switch to a second level of automation among levels of automation associated with the surgical task. The instruction may be based on the detection of a trigger. The device may operate at a second level of automation among levels of automation associated with the surgical task based on the instruction. The instruction may be based on monitoring the execution of the first level of automation. The execution of the first level of automation may be based on real-time surgical data.

[0193] Autonomous decision-making and assistance may be provided. The automatic determination of a level of autonomy (e.g., a level of automation) from a set of predetermined options may be based on identifying the situation of the surgical task (e.g., using situation awareness).

[0194] The determination of the level of automation of a smart medical device may be based on the monitored situation of the procedure. For example, a powered adaptable medical device control algorithm for controlling a medical device function may include a variable scale of automation. The monitoring of the instrument, the surgeon, and / or the patient may control the scale or level of automation (e.g., the control algorithm) without direct user control. In an example, the level of automation may be based on one or more of the capabilities of the device, the connection of the device to other devices, the presence of multiple personnel, or the detection of aspects of its status and / or configuration.

[0195] An automated task classification (e.g., a selectable set of available options) may be requested by a surgical instrument (e.g., a request message may be sent to a surgical hub), and the complexity assessment may be used, for example, to automatically determine the appropriate level of automation of the surgical instrument for performing the surgical task.

[0196] The functions described in this specification can bring about more autonomy. For example, there may be a release from automated operations based on during the service of the product. The level of autonomy can be determined based on the skill level of the user (e.g., a surgeon). For example, a less experienced user (e.g., a resident surgeon) may trigger a lower level of automation to be used, for example, to ensure the proper operation of surgical instruments. For example, a less experienced user (e.g., a resident surgeon) may have to pull the end cutter in an emergency situation of a surgical task. The surgical hub and / or surgical instrument may recognize that the user's skill level is inexperienced (e.g., new), minimize selections, and set the automation to maximize usability with limited user control activation. The level of autonomy may be associated with user-selectable options, and a less experienced user may have fewer selectable options than an experienced user (e.g., the selectable options are locked out and automatically grayed out). More advanced users may be given the option to automate more (e.g., surgical tasks resulting from using a higher level of automation) than less experienced users.

[0197] The level of autonomy can be based on the user's field of view. For example, when a surgical procedure moves outside the field of view (e.g., the laparoscopic field of view), the autonomous instrument control can be adjusted. For example, during surgery, the surgeon may have multiple ports linked to multiple different instruments to perform the intended task and may have one laparoscopic camera with a limited / defined field of view. Throughout the surgical procedure, the surgeon can switch between instruments and / or move the instruments to increase access. In such cases, one or more of the instruments may no longer be within the field of view. The instrument may activate an autonomous mode where the device function is not activated until it returns within the field of view.

[0198] The orientation of a device (e.g., a surgical instrument) can be autonomously controlled. For example, if the instrument is out of the field of view and the surgeon attempts to position it back into the field of view, the device end effector can autonomously control itself to rotate and / or reorient itself so that it does not contact other structures / tissues / organs until it is back in the field of view, which can prevent unintended actions.

[0199] Autonomous instrument control can be based on the surgeon's focus or field of view angle that can be fixed to a part of the screen. For example, during a surgery, the surgeon may be looking through a laparoscope camera. The surgeon can lose the sense of direction and / or be fixed for a particular task of the function (e.g., while mobilizing or creating access to a target site, the surgeon's focus can be fixed to a part of the screen). In such cases, the surgical instrument may switch to an autonomous mode that restricts a particular function until the surgeon's eyes are redirected towards the instrument. In such cases, the instrument can determine an autonomous mode that does not result in contact with other structures. For example, the autonomous mode can enable the end effector to move or reorient itself autonomously when the surgeon translates the device within the focus area of the screen.

[0200] The facility operator and / or the surgeon may choose to limit the autonomous customization based on the risk level. The level of autonomy for assisting with the surgery can be set / regulated (e.g., preemptively). For example, this can be based on administrative approvals, safety concerns and / or risks to the patient. The facility operator may choose to use the default automation, which can enable less experienced surgeons to obtain more reproducible results. The facility operator may choose to invalidate or deactivate the available levels of autonomous operation, for example, until it is verified that their behavior is consistent with the surgical intervention and / or their approach to the outcome. This can prevent the device from being brought into the facility and causing adverse results due to incompatible autonomous functions being used.

[0201] Adjusting the level of autonomy can be based on the task and / or patient risk. The laparoscopic ultrasound device can have multiple functions such as one or more of coagulation, cutting, incision, or grasping. Based on the intended function, the targeted zone, and / or the surgical task, the level of autonomy can be controlled by the intended function and / or patient risk, for example, by controlling the actions permitted by a combination of an autonomous system and / or human activation. In an example, the higher the risk to the patient, the more the execution of the task can be shifted from an autonomous state to a mixed state, and the surgeon needs to approve the action before it is performed, because the action may result in more negative consequences for the patient. In such a case, the surgeon can be prepared to react if an unintended result occurs (for example, a computer control system can have more precise control, but a human / surgeon can anticipate problems and adjust their response to complete the task). For example, the patient risk can be level 1. The surgical instrument can be used for grasping and tissue manipulation. Based on the level 1 risk, the surgical instrument can enable full autonomy of the jaws' opening and closing. For example, the patient risk can be level 2. The surgical instrument can be used for cutting and incision. Based on the level 2 risk, the surgical instrument can enable full autonomy and energy activation of the jaws when on non-vascular tissue but not on adipose tissue. In an example, the patient risk level can be 3. The surgical instrument can be used for vessel sealing and / or coagulation. Based on the level 3 risk level, the surgical instrument can enable full autonomy of the jaws, but the energy activation can be controlled / applied by the surgeon. This can result in less automation and more discrete movements. The level of autonomy can be based on the complexity of the procedure and / or patient-specific factors (such as co-morbidities). The level of autonomy can be based on detected problems.

[0202] The facility can limit the device automation of the process (e.g., the level of automation) until, for example, the facility has verified the use (e.g., the device function is intended and cost-beneficial).

[0203] The autonomy level can be based on (e.g., switched based on) safety risks, inaccurate operation of the surgical instrument, and / or detected problems related to the surgical instrument. For example, safe energy activation can be used. For ultrasonic and / or RF energy devices. Keeping the joe and the active electrode clean throughout the procedure, free of debris, can prevent tissue accumulation that can lead to unintended generator errors and / or performance degradation. To complete the task, the surgeon can remove the device from the patient, and the surgeon and / or instrument nurse can use a sponge to clean the joe. As a safety measure, the device may automatically activate the safety mode when it senses that the joe is being cleaned to ensure that energy activation cannot be initiated during cleaning. In an example, in the case of a harmonic device, if the blade is accidentally activated while using hemostasis while cleaning the blade, and / or if the blade comes into contact with something while being cleaned and / or while moving back to the patient, this can lead to scratches, cuts, and / or notches in the blade, which can increase the likelihood of early blade failure. These problems can be minimized by automatically deactivating the power while the device is outside the patient's body. For example, a safe joe closure can be used. For surgery, a device having an end effector can be closed while passing through a trocar. Automatically, the device may close the joe before being removed or inserted into the trocar. Thermal damage may be used. Energy devices such as harmonic scalpels (e.g., ACE family) can reach temperatures exceeding 200°C while performing the intended task or after deactivating the energy button. These devices may take longer to cool. Automatically, the system can control and / or limit the allowable movement of the joe until the allowable temperature of the joe is met to ensure that adjacent tissue is not inadvertently affected by thermal damage.

[0204] Having selectable levels of automation can result in more or less autonomy, based on the available settings and / or options. A facility operator and / or user may use more advanced features (e.g., associated with a higher level of autonomy) in certain settings. For example, smart software modules, more comprehensive control programs, and / or hardware capabilities can be used to determine the level of autonomy. Users of more advanced features may select a more restricted level of autonomy based on their needs (e.g., an educational facility may use a higher level of autonomy to ensure the safety of less experienced surgeons, or a regional facility may deactivate autonomous features because they are seen as not providing a benefit to patients in that region or may cause more problems than they solve).

[0205] A stepwise approach may be used to assess the risks associated with a surgical task, which may include one or more of accepting the risk, avoiding the risk, transferring the risk, or reducing the risk.

[0206] The device can be identified within a larger digital ecosystem. The device may be detected by other systems that are capable of integrating or communicating with the device and / or system.

[0207] The level of autonomy can be based on the presence of collaborative devices or systems. For example, a first smart device may detect the presence of a second smart device or surgical hub that can initiate an automated communication interaction. When the smart devices communicate, the type and / or configuration of the second smart hub may enable automatic updating or communication of operating parameters for the first device that can enable system updates. When the first device is operated, the first device can automatically update the second smart hub with information regarding its use, and this information can be automatically compiled, analyzed, and distributed to other systems that use certain portions of that data. This distributed data can be used when determining the level of autonomy of surgical instruments within the collaborative system. Product detection can be used as a trigger for performing the functions described herein. For example, the presence of a smart device in the vicinity of a smart hub, the presence of a smart component (e.g., RFID within a smart stapler) within a smart device, the presence or vicinity of a collaborative-compatible device (e.g., a smart proximity sensing scope or scope add-on, and the presence of a smart device having an integrated fiducial marker), a first hub within the range of a second hub, a smart OR, a network gateway, and / or a communication backplane generator, or a first imaging system within the range of a second imaging system, one or more of which may be a trigger.

[0208] The ability of a system to measure and / or detect information to operate at a level of autonomy can be determined.

[0209] An HCP within the room that can interact or operate with the device may be determined. For example, a sufficient number of appropriate users may be determined, which may include one or more of the area of expertise and / or duties (e.g., surgeon, anesthesiologist, operating room technician, cardiovascular nurse, etc.), level of experience (e.g., full-time within the medical field, time / amount of work within a given specialty, time / amount of work within a given surgery, time / amount of work using a particular device, and / or certification and training), skill level (e.g., detected by previous operations, device use, results, etc.), or what is manually entered into the system by the user / administrator. To determine who is in the room, one or more of manual input, manual badge scanners, room badge scanners (e.g., that automatically check and update who is in the room), face recognition, etc. may be used.

[0210] The level of autonomy may be based on one or more of whether the number of people in the room is insufficient to default to automated operations by a limited number of users. For example, if the number of staff present is insufficient, the level of automation may be determined (e.g., adjusted to complete the procedure). In an example, the surgery (e.g., surgical task) may be updated, which may result in higher efficiency as automation increases with the adjustment.

[0211] Notification of the reason for selecting a particular level of automation may be provided, which may be used to change the selection criteria used when selecting the level of automation. The user may be notified of the variables affecting the level of automation. For example, if there are not enough personnel in the room, the surgical instruments may be default set to manual operation (e.g., or default set to autonomous).

[0212] The determination of the magnitude of the trigger and / or autonomy can be based on the monitored and / or calculated data feed. During the execution of a task that is being executed autonomously, there may be suboptimal autonomy level options. In such a case, it can be determined how to adjust the level of that autonomy. For example, it can be determined to autonomously and safely change from one level of autonomy to a lower level. In an example, it can be determined to end and stop the current task (e.g., not proceed to the next task). In an example, it can be determined to end immediately.

[0213] The detected magnitude of an error in information can be used as a means for determining the level of autonomy. For example, as the level of error increases, the level of autonomy decreases. The magnitude of the error can include system-based cumulative total errors and / or the thresholds of single errors. In an example, a single error with a high threshold can reduce the autonomy of the system more than a plurality of errors of small or negligible magnitude. There can be errors that partially remove the autonomy of the system. For example, a highly autonomous device system such as a surgical robot can have a plurality of arms connected thereto. During a startup test sequence, one of the arms can be detected to have an error with its feedback sensor based on output data and feedback. It can be determined that the function of the arm is not operating correctly. The system can exclude the autonomous activity of that arm while maintaining autonomous activity with other arms. There can be errors that completely remove the autonomy of the system. For example, a highly autonomous device system such as a surgical robot can use a mainframe processing location. There may be redundancy in certain aspects (e.g., power supply), but there may be no redundancy in the central processor. If there is a failure in the processor, all autonomy of the system can be removed. There may be errors that do not affect the level of autonomy of the system. For example, a highly autonomous system such as a surgical robot can rely on GPS signals to calculate time, date, and the country of location for record-keeping. The GPS signal can be lost and can generate an error. There can be a 24-hour holdover period during which previous data is still considered accurate. In such a case, even if an error occurs, there may be no impact on functionality or the level of autonomy.

[0214] In an example, the true cumulative error may not need to be calculated. In such a case, specific functions and capabilities related to autonomy can be removed according to the associated failure modes. Each potential autonomous function of the device can be mapped to one or more corresponding physical / software functions of the device.

[0215] A user control input actuator may be provided. The self-regulatory stop switch may be, for example, a button that removes all autonomy from the system regardless of the current step or state of the system, e.g., a button that removes autonomy from the system after the system has completed its current activity, a plurality of state buttons (e.g., state 1 for the button allows autonomy to end its current activity, and state 2 for the button immediately removes all autonomy regardless of the current activity), or a configurable button (e.g., the button can be configured as to how it should operate, such as removing autonomy immediately or upon completion of an activity).

[0216] Sensing events associated with the device may be used, which may include one or more of initial sensing force contact, rate of increase of force, impedance threshold, heat dissipation / damage, or Joule temperature energy events. For example, when Joe is closed, a first timing at which current through the motor can be sensed may be used to determine tissue height. If the end effector is determined to be thick, the motor speed may be changed based on that threshold. The trigger may be used to cause a change in autonomy. For example, an overly thick state may cause the device to stop automation and require user input.

[0217] Impedance may be used as a trigger. For example, when energy is activated within the joe, the generator can monitor the impedance of the tissue and / or blood vessel to determine when to turn off the energy. The product code may be indicated for the maximum vessel occlusion size (e.g., typically ≤5 mm or ≤7 mm size). The energy algorithm may take into account the impedance over time. If the energy activation cycle is too short compared to the normal cycle, this can indicate and / or signal damaged and / or diseased tissue and notify the user to take over the activation control. If the energy activation cycle is longer than the normal cycle, this can indicate and / or signal that the tissue is fatty and / or that the vessel size is larger than what the product is indicated for, and the responsibility can be transferred to the surgeon. Tissue accumulated on the joe can change the impedance control. When this condition occurs based on the calculated time versus the actual time, the user can be notified to clean the joe. In an example, harmonic 7 may operate at power level 3 only for vessels with a diameter of 5 mm or less, and the forward hemostasis mode may operate for vessel sizes of 7 mm or less. In such cases, the system can autonomously select / change the power level based on the identified vessel size. When the vessel is above or between sizes, the system may request from the user which power level to continue with.

[0218] Thermal diffusion can be used as a trigger. During energy activation and vessel occlusion, criteria that can be monitored along with hemostasis are thermal damage and / or thermal diffusion. Temperature and / or lateral thermal damage can be considerations for the surgeon when using energy-based techniques. The surgeon may be concerned about damage to nearby structures either by direct contact or by visually unrecognizable transmission of energy. The surgeon may be concerned about the potential impact of tissue damage on the inflammatory response and the patient's overall recovery. Monitoring thermal diffusion during energy activation, along with tissue type, thickness, power level, and / or clamp pressure, can lead to calculating the actual birth nominal, which can be used to notify the user and / or to autonomously adjust the power setting and / or clamp pressure (e.g., harmonic ACE +72.54+ / -0.48mm. deviation referring to mean and standard). Preclinical comparisons of goat vessel occlusion can be used when autonomously adjusting. Thermal damage can result from heat generation by high-energy devices and can be an important component for vessel occlusion and / or tissue cutting. The temperature reached by the instrument can depend on multiple variables including tissue type, tissue thickness, energy used, and / or power setting. High-energy devices can reach an instrument temperature of at least 100°C during activation on tissue. There can be situations where the temperature of an ultrasonic device reaches beyond the range of 100°C due to tissue conditions. When Joe reaches this temperature, the surgeon may accidentally come into contact with unintended tissue and cause trauma. When this occurs, a waiting time can be triggered until the Joe temperature drops to an acceptable temperature. In such a case, if contact is made with tissue where no trauma would occur, or if the surgeon attempts to move the device, the autonomous control may take over limited control (e.g., if the surgeon moves the device distally with a faster and / or larger motion than allowed, the autonomy may decelerate and / or reduce the displacement to ensure that the hot Joe cannot come into contact with the tissue and / or the unintended treatment area).

[0219] The stapling device can be used as a trigger. For example, an unexpected joint movement force can be used.

[0220] An automated risk determination level may be provided. User selections may be compared to benchmarks regarding other facility users or global user selection levels to determine a permission level associated with the level of autonomy. The dynamic selection of the surgeon can be monitored to control autonomous selection. The setting of an uncertainty level or a preselected user risk level may be used to determine an acceptable risk level in that determination. Previous operations or results may be used to determine what the effective risk level may be.

[0221] The system can provide a risk matrix used to determine an appropriate risk. The matrix may include medical or patient claim histories (such as International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM), Hierarchical Condition Categories (HCC), electronic health records (HER - health information technology (HIT) database and / or equivalents)).

[0222] The overall risk determination may be a combination of risks associated with the patient, device, and procedure.

[0223] The adjustment of the level of the autonomous function may be based on past user control, previous autonomous operations, and / or past results resulting from previous autonomous decisions. Autonomous failures and / or user overrides / supports may reduce the level of autonomy and / or the frequency of proposed autonomous operations. Undesirable results may adjust the level of autonomous engagement (e.g., fewer autonomous engagements if a user error caused it compared to if an autonomous operation caused it). Past user interactions that have caused confusion, misuse, or usage explanation problems may result in the system assisting with autonomous prompts and / or controls to minimize future problems with usage, such as delays in procedure counts.

[0224] Data aggregation can be utilized to adjust the level of autonomy. For example, weighted comparisons can be used to determine how to react to past information. Criticality of failures, local frequencies, uniqueness of failures, etc. can be used. Uniqueness can include complete motor failure and / or a complete reset of the system (e.g., RF interference with system operation that results in a complete restart to continue use).

[0225] The following is a numbered list of embodiments, which may or may not be claimed. 1. A device for changing the level of automation of a surgical instrument for performing a surgical procedure, the device comprising a processor, the processor being configured to receive an indication of a surgical procedure to be performed using the surgical instrument, the surgical procedure including a plurality of surgical tasks to be performed using the surgical instrument, monitor the execution of one of the plurality of surgical tasks while the surgical instrument is operating at a first level of automation, detect a trigger event associated with the execution of the surgical task, and based on the trigger event, switch the operation of the surgical instrument from the first level of automation to a second level of automation.

[0226] Advantageously, a device for changing the level of automation of a surgical instrument for performing a surgical procedure increases or decreases the level of automation based on the monitored execution of the surgical tasks of the procedure to reduce the risk of adverse surgical outcomes for that task or procedure, and thus ultimately improve the surgical outcome for the surgical procedure being performed.

[0227] As an example, the results of tissue resection can correspond to the integrity of the sealing line, and a seal with less leakage is equivalent to a more positive result than a seal with more leakage. The results of the procedure can also correspond to the number of intraoperative complications that can include, for example, device malfunction, leakage of the sealing line, misfiring of the staple line, and the like.

[0228] 2. The level of automation is the device according to embodiment 1, corresponding to the number of surgical tasks from a plurality of automated surgical tasks.

[0229] By changing the number of automated surgical tasks, it is possible to reduce the risk to the patient and the risk of adverse surgical outcomes.

[0230] 3. The second level of automation is the device according to embodiment 1 or embodiment 2, associated with automating fewer surgical tasks when compared to the first level of automation.

[0231] Accordingly, switching the level of automation from the first level to the second level results in one or more surgical tasks that were autonomously performed before the switch and are manually performed after the switch.

[0232] 4. Automating a surgical task is the device according to embodiment 2 or embodiment 3, associated with zero manual input from the surgeon.

[0233] 5. Automating a surgical task is the device according to embodiment 2 or embodiment 3, associated with a reduction in manual input from the surgeon.

[0234] 6. The execution of a surgical task is the device according to any one of embodiments 1 to 5, based on real-time surgical data.

[0235] By monitoring the execution of surgical tasks using real-time surgical data, it is possible to detect trigger events in real-time and respond thereto, for example, by changing whether the monitored surgical task is automated and / or by changing whether one or more of the subsequent surgical tasks of that surgical procedure are automated, enabling the level of automation to be changed during the current surgical procedure, which can increase the likelihood of a positive surgical outcome for that surgical procedure.

[0236] 7. The real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, and task data, and is the device according to Embodiment 6.

[0237] For example, the surgical instrument data may include the measured physical state from a surgical device performing a surgical task. The user data may include the measured physiological state of the patient, surgeon, and / or staff participating in the surgery.

[0238] 8. The processor is further configured to detect a trigger event by comparing the execution of the surgical task with a trigger event threshold value, and is the device according to any one of Embodiments 1 to 7.

[0239] 9. Monitoring the execution of a surgical task includes comparing one or more of the real-time surgical data with their respective ideal surgical data, and detecting a trigger event when the comparison exceeds a trigger event threshold value, and is the device according to one of Embodiments 6, 7, or 8.

[0240] 10. Each ideal surgical data is generated from past data for that surgical task, and is the device according to Embodiment 9.

[0241] Ideal surgical data can be generated from past surgical data for surgical tasks associated with positive or ideal outcomes. If current surgical data deviates from past surgical data for surgical tasks associated with positive outcomes, it indicates that the surgical outcome for the current surgery may have a lower success rate / positivity than the outcomes associated with past surgical tasks. It may be possible to improve the likelihood of a successful surgical outcome by responding to trigger events by changing the level of automation, which may include changing whether the current task and / or subsequent tasks are automated.

[0242] 11. The surgical procedure includes excising tissue, and the plurality of surgical tasks includes one or more of control of an energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, reload alignment slot management, for the device according to any of embodiments 1 to 10.

[0243] 12. The surgical device comprises a memory storing a plurality of surgical tasks associated with a surgical procedure and surgical tasks automated for each level of autonomy, for the device according to any of embodiments 1 to 11.

[0244] 13. The surgical instrument is suitable for performing two or more surgical procedures, the surgical instrument performs a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure, the surgical device comprises a memory storing a first plurality of surgical tasks associated with the first surgical procedure and a second plurality of surgical tasks associated with the second surgical procedure, the memory further storing an association between the surgical tasks automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks, for the device according to embodiment 12.

[0245] The surgical instrument can be used for a plurality of procedures each including a different plurality of surgical tasks, and the device can be configured to change the level of automation of the instrument to perform any one of these procedures.

[0246] 14. The device according to any one of embodiments 1 to 13, wherein the device is a surgical hub.

[0247] 15. A computer-implemented method for providing a command or recommendation for changing the level of automation of a surgical instrument for performing a surgical operation, receiving an instruction for a surgical operation to be performed using the surgical instrument, the surgical operation including a plurality of surgical tasks to be performed using the surgical instrument, monitoring the execution of a surgical task among the plurality of surgical tasks while the surgical instrument is operating at a first level of automation among a plurality of levels of automation, detecting a trigger event associated with the execution of the surgical task, providing a command to the surgical instrument to switch the operation of the surgical instrument from the first level of automation to a second level of automation or providing a recommendation to the user based on the trigger event.

[0248] Advantageously, the computer-implemented method monitors the execution of surgical tasks performed by the instrument during a surgical operation and, based on the trigger event, can provide a command to the instrument or a recommendation to the user to change the level of automation of the surgical instrument for performing the surgical operation, increasing or decreasing the level of automation to reduce the risk of adverse surgical outcomes for that procedure and thus ultimately improving the surgical outcome for the surgical operation being performed.

[0249] As an example, the results of tissue resection can correspond to the integrity of the sealing line, and a seal with less leakage is equivalent to a more positive result than a seal with more leakage. The results of the procedure can also correspond to the number of intraoperative complications that can occur, including, for example, instrument malfunction, leakage of the sealing line, misfiring of the staple line, and the like.

[0250] 16. The method of embodiment 15, wherein the level of automation corresponds to the number of surgical tasks from a plurality of surgical tasks being automated.

[0251] 17. The method of embodiment 16, wherein a second level of automation is associated with automating fewer surgical tasks as compared to the first level of automation.

[0252] 18. The method according to any one of embodiments 15 to 17, wherein monitoring the execution of a surgical task is based on real-time surgical data.

[0253] By monitoring the execution of a surgical task using real-time surgical data, it is possible to detect and respond to trigger events in real time, for example, by changing whether the monitored surgical task is automated and / or by changing whether one or more of the subsequent surgical tasks of that surgical procedure are automated, which can enable the level of automation to be changed during the current surgical procedure, which can increase the likelihood of a positive surgical outcome for that surgical procedure.

[0254] 19. The method of embodiment 18, wherein the real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, and task data.

[0255] For example, the surgical instrument data may include the measured physical state from a surgical device performing a surgical task. The user data may include the measured physiological state of the patient, surgeon, and / or staff participating in the surgery.

[0256] 20. Monitoring the execution of a surgical task includes comparing one or more of the real-time surgical data with their respective ideal surgical data and detecting that a trigger event occurs when the comparison exceeds a threshold, according to the method described in Embodiment 18 or Embodiment 19.

[0257] 21. According to the method described in Embodiment 20, each ideal surgical data is generated from past data of the surgical task.

[0258] Ideal surgical data can be generated from past surgical data for surgical tasks associated with positive or ideal outcomes. If the current surgical data deviates from past surgical data for surgical tasks associated with positive outcomes, it indicates that the surgical outcome for the current surgery may have a lower success rate / positivity than the outcomes associated with past surgical tasks. It may be possible to improve the likelihood of a successful surgical outcome by changing the level of automation in response to a trigger event, which may include changing whether the current task and / or subsequent tasks are automated.

[0259] 22. The method according to Embodiment 21 further includes generating each ideal surgical data.

[0260] 23. The method according to any one of Embodiments 15 to 22 includes retrieving from the memory a plurality of surgical tasks associated with the surgery and the surgical tasks automated for each level of autonomy.

[0261] 24. The surgical instrument is suitable for performing two or more surgical procedures, the surgical instrument performs a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure, and the method includes retrieving, from a memory, instructions for the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and retrieving, from the memory, an association between automated surgical tasks for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks, the method according to embodiment 23.

[0262] The surgical instrument can be used for a plurality of procedures each including a different plurality of surgical tasks, and the device can be configured to change the level of automation of the instrument to perform any one of these procedures.

[0263] 25. A computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of embodiments 15 to 24.

[0264] 26. A computer-readable medium including instructions that, when executed by a computer, cause the computer to perform the method according to any one of embodiments 15 to 24.

[0265] 27. A device for performing a plurality of surgical tasks in a surgical procedure, the device comprising a processor, the processor operating at a first level of automation, obtaining an instruction to switch to a second level of automation, the instruction being based on detecting a trigger, and operating at the second level of automation based on the instruction, the device being configured to perform.

[0266] Advantageously, a device that can operate at different levels of automation enables changes in automation and enables the procedure to be more or less automated, reducing the risk of adverse surgical outcomes for that procedure, and thus ultimately improving the surgical outcome for the surgical procedure being performed.

[0267] 28. The device according to embodiment 27, wherein the first level of automation and the second level of automation differ in the number of surgical tasks of the surgical procedure to be automated.

[0268] 29. The device according to embodiment 27 or 28, wherein the instruction is based on monitoring the execution of one of the plurality of surgical tasks at the first level of automation.

[0269] 30. The device according to embodiment 29, wherein the execution of the surgical task at the first level of automation is based on real-time surgical data.

[0270] By using real-time surgical data to monitor the execution of surgical tasks, it is possible to detect and respond to trigger events in real time, for example, by changing whether the monitored surgical task is automated and / or by changing whether one or more of the subsequent surgical tasks of that surgical procedure are automated, enabling the level of automation to be changed during the current surgical procedure, which can increase the likelihood of a positive surgical outcome for that surgical procedure.

[0271] The following are numbered aspects of the present disclosure that may or may not be claimed. 1. A device for automating a surgical task, the device comprising a processor, the processor being receiving an instruction for a surgical task to be performed using a surgical instrument, wherein the capabilities of the surgical instrument are associated with a plurality of levels of automation, monitoring the execution of a surgical task while the surgical instrument is operating at a first level of automation among a plurality of levels of automation; detecting a trigger event associated with the execution of the surgical task; and switching the operation of the surgical instrument from a first level of automation to a second level of automation among the plurality of levels of automation based on the trigger event, a device configured to perform. 2. The device according to aspect 1, wherein the capabilities of the surgical instrument include a set of surgical instrument tasks, and the level of automation is associated with automating one or more surgical instrument tasks from the set of surgical instrument tasks. 3. The device according to aspect 2, wherein automating one or more surgical tasks is associated with a manual input of 0 from the surgeon. 4. The device according to aspect 2, wherein automating one or more surgical tasks is associated with a reduction in manual input from the surgeon. 5. The device according to aspect 2, wherein the execution of the surgical task is based on real-time surgical data. 6. The device according to aspect 5, wherein the real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, task data, or past data. 7. The processor is detecting a trigger event by comparing the execution of the surgical task with a trigger event threshold; and switching the operation of the surgical instrument from a first level of automation to a second level of automation based on the trigger event, the second level of automation being associated with automating fewer surgical instrument tasks when compared to the first level of automation, and further configured to perform. The device according to aspect 2. 8. The processor is detecting a trigger event by comparing the execution of the surgical task with a trigger event threshold; Based on a trigger event, switch the operation of the surgical instrument from a first level of automation to a second level of automation, and the second level of automation is associated with a higher manual input from the surgeon when compared to the first level of automation, and the device according to aspect 1 is further configured to perform the above. 9. Monitoring the execution of a surgical task includes comparing one or more of the real-time surgical data with their respective ideal surgical data, for the device according to aspect 6. 10. A method for automating a surgical task, the method comprising: Receiving an instruction for a surgical task to be performed using a surgical instrument, wherein the capabilities of the surgical instrument are associated with multiple levels of automation; Monitoring the execution of the surgical task while the surgical instrument is operating at a first level of automation among the multiple levels of automation; Detecting a trigger event associated with the execution of the surgical task; Based on the trigger event, switching the operation of the surgical instrument from a first level of automation to a second level of automation among the multiple levels of automation. 11. The method according to aspect 10, wherein the capabilities of the surgical instrument include a set of surgical instrument tasks, and the level of automation is associated with automating one or more surgical instrument tasks from the set of surgical instrument tasks. 12. Automating one or more surgical tasks is associated with a manual input of 0 from the surgeon, for the method according to aspect 11. 13. Automating one or more surgical tasks is associated with a reduction in manual input from the surgeon, for the method according to aspect 11. 14. The execution of the surgical task is based on real-time surgical data, for the method according to aspect 11. 15. The real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, task data, or past data, for the method according to aspect 14. 16. Detecting a trigger event by comparing the execution of a surgical task with a trigger event threshold, and Based on the trigger event, switching the operation of the surgical instrument from a first level of automation to a second level of automation, wherein the second level of automation is associated with automating fewer surgical instrument tasks when compared to the first level of automation, the method according to aspect 11, further comprising. 17. Detecting a trigger event by comparing the execution of a surgical task with a trigger event threshold, and Based on the trigger event, switching the operation of the surgical instrument from a first level of automation to a second level of automation, wherein the second level of automation is associated with a higher manual input from the surgeon when compared to the first level of automation, the method according to aspect 10, further comprising. 18. Monitoring the execution of a surgical task includes comparing one or more of the real-time surgical data with their respective ideal surgical data, the method according to aspect 15. 19. A device for automating a surgical task, the device comprising A processor, the processor being Operating at a first level of automation among a plurality of levels of automation associated with the surgical task, and Obtaining an instruction for switching to a second level of automation among a plurality of levels of automation associated with the surgical task, the instruction being based on detecting a trigger, and Based on the instruction, operating at a second level of automation among a plurality of levels of automation associated with the surgical task, a device configured to perform. 20. The instruction is based on monitoring the execution of the first level of automation, the device according to aspect 19. 21. The execution of the first level of automation is based on real-time surgical data, the device according to aspect 20.

[0272] 〔Embodiment〕 (1) A device for changing the level of automation of a surgical instrument for performing a surgical operation, the device comprising: a processor, the processor being configured to: receive an instruction for a surgical operation to be performed using the surgical instrument, the surgical operation including a plurality of surgical tasks to be performed using the surgical instrument; monitor the execution of one of the plurality of surgical tasks while the surgical instrument is operating at a first level of automation; detect a trigger event associated with the execution of the surgical task; based on the trigger event, switch the operation of the surgical instrument from the first level of automation to a second level of automation. (2) The device according to embodiment 1, wherein the level of automation corresponds to the number of surgical tasks from the plurality of surgical tasks to be automated. (3) The device according to embodiment 1 or embodiment 2, wherein the second level of automation is associated with automating fewer surgical tasks when compared to the first level of automation. (4) The device according to embodiment 2 or embodiment 3, wherein automating a surgical task is associated with zero manual input from a surgeon. (5) The device according to embodiment 2 or embodiment 3, wherein automating a surgical task is associated with a reduction in manual input from a surgeon.

[0273] (6) The device according to any one of embodiments 1 to 5, wherein the execution of the surgical task is based on real-time surgical data. (7) The device according to embodiment 6, wherein the real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, and task data. (8) The device according to any one of embodiments 1 to 7, wherein the processor is further configured to detect the trigger event by comparing the execution of the surgical task with a trigger event threshold value. (9) Monitoring the execution of the surgical task includes comparing one or more of the real-time surgical data with respective ideal surgical data, and detecting that a trigger event occurs when the comparison exceeds a trigger event threshold value. The device according to any one of embodiments 6, 7, or 8. (10) The device according to embodiment 9, wherein the respective ideal surgical data is generated from past data of the surgical task.

[0274] (11) The surgical operation includes excising tissue, and the plurality of surgical tasks includes one or more of energy source control, cutting, stapling, knob orientation, body orientation, body position, anvil joe force, and reload alignment slot management. The device according to any one of embodiments 1 to 10. (12) The surgical device according to any one of embodiments 1 to 11, comprising a memory that stores the plurality of surgical tasks associated with the surgical operation and the surgical tasks automated for each level of autonomy. (13) The surgical instrument is suitable for performing two or more surgical operations. The surgical instrument performs a first plurality of surgical tasks for the first surgical operation and a second plurality of surgical tasks for the second surgical operation. The surgical device includes a memory that stores the first plurality of surgical tasks associated with the first surgical operation and the second plurality of surgical tasks associated with the second surgical operation. The memory further stores an association between the surgical tasks automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks. The device according to embodiment 12. (14) The device according to any one of embodiments 1 to 13, wherein the device is a surgical hub. (15) A computer-implemented method for providing a command or recommendation for changing the level of automation of a surgical instrument for performing a surgical operation, the method comprising: receiving an instruction for a surgical operation to be performed using the surgical instrument, the surgical operation including a plurality of surgical tasks to be performed using the surgical instrument; monitoring the execution of a surgical task among the plurality of surgical tasks while the surgical instrument is operating at a first level of automation among the plurality of levels of automation; detecting a trigger event associated with the execution of the surgical task; providing a command to the surgical instrument or providing a recommendation to a user for switching the operation of the surgical instrument from the first level of automation to a second level of automation based on the trigger event.

[0275] (16) The method according to embodiment 15, wherein the level of automation corresponds to the number of surgical tasks from the plurality of automated surgical tasks. (17) The method according to embodiment 16, wherein the second level of automation is associated with automating fewer surgical tasks when compared to the first level of automation. (18) The method according to any one of embodiments 15 to 17, wherein monitoring the execution of the surgical task is based on real-time surgical data. (19) The method according to embodiment 18, wherein the real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, and task data. (20) The method according to embodiment 18 or 19, wherein monitoring the execution of the surgical task includes comparing one or more of the real-time surgical data with respective ideal surgical data and detecting that a trigger event occurs when the comparison exceeds a threshold.

[0276] (21) Each of the respective ideal surgical data is generated from past data of the surgical task, according to the method described in Embodiment 20. (22) The method according to Embodiment 21, further comprising generating each of the respective ideal surgical data. (23) The method according to any one of Embodiments 15 to 22, comprising retrieving from a memory the plurality of surgical tasks associated with the surgical operation and the surgical tasks automated for each level of autonomy. (24) The surgical instrument is suitable for performing two or more surgical operations. The surgical instrument performs a first plurality of surgical tasks for the first surgical operation and a second plurality of surgical tasks for the second surgical operation. The method comprises retrieving from the memory instructions for the first plurality of surgical tasks associated with the first surgical operation and the second plurality of surgical tasks associated with the second surgical operation, and retrieving from the memory an association between the surgical tasks automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks, according to the method described in Embodiment 23. (25) A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of Embodiments 15 to 24.

[0277] (26) A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of Embodiments 15 to 24. (27) A device for performing a plurality of surgical tasks in a surgical operation, the device comprising a processor, the processor operating at a first level of automation, and obtaining an instruction for switching to a second level of automation, the instruction being based on detecting a trigger. A device configured to operate at the second automation level based on the indication. (28) The device according to embodiment 27, wherein the first automation level and the second automation level differ in the number of surgical tasks of the surgical procedure to be automated. (29) The device according to embodiment 27 or embodiment 28, wherein the indication is based on monitoring the execution of one of the plurality of surgical tasks at the first automation level. (30) The device according to embodiment 29, wherein the execution of the surgical task at the first automation level is based on real-time surgical data.

Claims

1. A device for changing the level of automation of a surgical instrument for performing a surgical operation, the device comprising: a processor, the processor being configured to: receive an instruction for a surgical operation to be performed using the surgical instrument, the surgical operation including a plurality of surgical tasks to be performed using the surgical instrument; monitor the execution of one of the plurality of surgical tasks while the surgical instrument is operating at a first level of automation; detect a trigger event associated with the execution of the surgical task; and based on the trigger event, switch the operation of the surgical instrument from the first level of automation to a second level of automation.

2. The device according to claim 1, wherein the level of automation corresponds to the number of surgical tasks from the plurality of surgical tasks to be automated.

3. The device according to claim 1 or claim 2, wherein the second level of automation is associated with automating fewer surgical tasks when compared to the first level of automation.

4. The device according to claim 2, wherein automating a surgical task is associated with zero manual input from a surgeon.

5. The device according to claim 2, wherein automating a surgical task is associated with a reduction in manual input from a surgeon.

6. The device according to claim 1, wherein the execution of the surgical task is based on real-time surgical data.

7. The device according to claim 6, wherein the real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, and task data.

8. The device according to claim 1, wherein the processor is further configured to detect the trigger event by comparing the execution of the surgical task with a trigger event threshold value. **Claim 9** Monitoring the execution of the surgical task includes comparing one or more of the real-time surgical data with respective ideal surgical data, and detecting that a trigger event occurs when the comparison exceeds a trigger event threshold value. The device according to claim 6. **Claim 10** The device according to claim 9, wherein the respective ideal surgical data is generated from past data of the surgical task. **Claim 11** The surgical operation includes excising tissue, and the plurality of surgical tasks includes one or more of control of an energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, and reload alignment slot management. The device according to claim 1. **Claim 12** The surgical device according to claim 1, comprising a memory that stores the plurality of surgical tasks associated with the surgical operation and the surgical tasks automated for each level of autonomy. **Claim 13** The surgical instrument is suitable for performing two or more surgical operations. The surgical instrument performs a first plurality of surgical tasks for the first surgical operation and a second plurality of surgical tasks for the second surgical operation. The surgical device includes a memory that stores the first plurality of surgical tasks associated with the first surgical operation and the second plurality of surgical tasks associated with the second surgical operation. The memory further stores an association between the surgical tasks automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks. The device according to claim 12. **Claim 14** The device according to claim 1, wherein the device is a surgical hub.

15. A computer-implemented method for providing a command or recommendation for changing the level of automation of a surgical instrument for performing a surgical operation, comprising: receiving an instruction for a surgical operation to be performed using a surgical instrument, the surgical operation including a plurality of surgical tasks to be performed using the surgical instrument; monitoring the execution of a surgical task among the plurality of surgical tasks while the surgical instrument is operating at a first level of automation among the plurality of levels of automation; detecting a trigger event associated with the execution of the surgical task; based on the trigger event, providing a command to the surgical instrument or providing a recommendation to the user to switch the operation of the surgical instrument from the first level of automation to a second level of automation.

16. The method according to claim 15, wherein the level of automation corresponds to the number of surgical tasks from the plurality of automated surgical tasks.

17. The method according to claim 16, wherein the second level of automation is associated with automating fewer surgical tasks when compared to the first level of automation.

18. The method according to any one of claims 15 to 17, wherein monitoring the execution of the surgical task is based on real-time surgical data.

19. The method according to claim 18, wherein the real-time surgical data includes at least one of user data, surgical environment data, surgical instrument data, and task data.

20. Monitoring the execution of the surgical task includes comparing one or more of the real-time surgical data with respective ideal surgical data, and detecting that a trigger event occurs when the comparison exceeds a threshold value. The method according to claim 18.

21. The respective ideal surgical data is generated from past data of the surgical task. The method according to claim 20.

22. The method further includes generating the respective ideal surgical data. The method according to claim 21.

23. The method includes retrieving from a memory the plurality of surgical tasks associated with the surgical procedure and the surgical tasks automated for each level of autonomy. The method according to claim 15.

24. The surgical instrument is suitable for performing two or more surgical procedures. The surgical instrument performs a first plurality of surgical tasks for the first surgical procedure and a second plurality of surgical tasks for the second surgical procedure. The method includes retrieving from the memory instructions for the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and retrieving from the memory an association between the surgical tasks automated for each level of autonomy for each of the first plurality of surgical tasks and the second plurality of surgical tasks. The method according to claim 23.

25. A computer program including instructions that, when executed by a computer, cause the computer to perform the method according to claim 15.

26. A computer-readable medium including instructions that, when executed by a computer, cause the computer to perform the method according to claim 15.

27. A device for performing a plurality of surgical tasks in a surgical operation, the device comprising: a processor, the processor being configured to: operate at a first level of automation; obtain an instruction to switch to a second level of automation, the instruction being based on detecting a trigger; and operate at the second level of automation based on the instruction. **Claim 28** The device according to claim 27, wherein the first level of automation and the second level of automation differ in the number of surgical tasks of the surgical operation to be automated. **Claim 29** The device according to claim 27 or claim 28, wherein the instruction is based on monitoring the execution of one of the plurality of surgical tasks at the first level of automation. **Claim 30** The device according to claim 29, wherein the execution of the surgical task at the first level of automation is based on real-time surgical data.

Images