Dynamically determining surgical autonomous levels

Abstract

Systems, methods, and instruments for automating surgical tasks are described herein. The device may receive an indication of a surgical task to be performed with a surgical instrument. The capabilities of the surgical instrument may be associated with a level of automation. The device may monitor performance of the surgical task if the surgical instrument operates at a first level of automation associated with the level of automation. The device may detect a trigger event associated with the performance of the surgical task and switch operation of the surgical instrument from the first automation level to a second automation level associated with the automation level, for example, based on the trigger event.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to the following concurrently filed patent applications, the contents of each of which are incorporated herein by reference:

[0003] U.S. Patent Application Titled “METHOD OF CONTROLLING AUTONOMOUSOPERATIONS IN A SURGICAL SYSTEM,” Attorney Docket No. END9430USNP1.

[0004] U.S. Patent Application Titled “DETECTING FAILURE MITIGATIONASSOCIATED WITH AUTONOMOUSSURGICAL TASK,” Attorney Docket No. END9430USNP3. Background Art

[0005] Surgical procedures are typically performed in a surgical suite or operating room at a medical facility, such as, for example, a hospital. Various surgical devices and systems are utilized in performing surgical procedures. In the digital and information age, medical systems and facilities are often slow to implement systems or procedures that utilize newer and improved technologies due to patient safety and a general desire to maintain traditional practices. Summary of the invention

[0006] A device for changing the automation level of a surgical instrument for performing a surgical procedure. The device includes a processor. The processor is configured to receive an indication of a surgical procedure to be performed using a surgical instrument, the surgical procedure including a plurality of surgical tasks to be performed using the surgical instrument. The processor is configured to monitor the execution of a surgical task among the plurality of surgical tasks when the surgical instrument is operated at a first automation level. The processor is further configured to detect a trigger event associated with the execution of the surgical task. The processor is configured to switch the operation of the surgical instrument from a first automation level to a second automation level based on the trigger event. Advantageously, the device for changing the automation level of a surgical instrument for performing a surgical procedure enables the automation level to be increased or decreased based on the monitored execution of the surgical task of the operation to reduce the risk of an adverse surgical outcome of the task or operation, and thus ultimately improve the surgical outcome of the surgical procedure performed. For example, the outcome of tissue resection may correspond to the integrity of the seal line, wherein less seal leakage is equivalent to a more positive outcome compared to more seal leakage. The outcome of the operation also corresponds to the number of difficulties that occurred during the surgical procedure, which difficulties may include, for example, instrument failure, seal line leakage, nail line misfiring, etc.

[0007] The level of automation may correspond to the number of automated surgical tasks in the plurality of surgical tasks. By varying the number of automated surgical tasks, the risk to the patient and the risk of adverse surgical outcomes may be reduced.

[0008] The second automation level may be associated with automating fewer surgical tasks when compared to the first automation level. Thus, switching the automation level from the first level to the second level causes one or more surgical tasks to be manually performed after the switch that were originally performed autonomously before the switch.

[0009] Automating surgical tasks can be associated with zero manual input from the surgeon.

[0010] Automating surgical tasks may be associated with reduced manual input from the surgeon.

[0011] The execution of surgical tasks can be based on real-time surgical data. By using real-time surgical data to monitor the execution of surgical tasks, trigger events can be detected and responded to in real time, making it possible to change the level of automation during the current surgical procedure, such as by changing whether the monitored surgical task is automated and / or changing whether one or more of the subsequent surgical tasks of the surgical procedure are automated, which can increase the chances of a positive surgical outcome for the surgical procedure.

[0012] Real-time surgical data may include at least one of: user data, surgical environment data, surgical instrument data, and task data. For example, surgical instrument data may include physical conditions measured from a surgical device performing a surgical task. User data may include measured physiological conditions of a patient, surgeon, and / or personnel involved in a surgical procedure.

[0013] The processor may be further configured to detect a trigger event by comparing the performance of the surgical task to a trigger event threshold.

[0014] Monitoring the performance of the surgical task may include comparing one or more of the real-time surgical data to corresponding ideal surgical data and detecting the occurrence of a trigger event when the comparison exceeds a trigger event threshold.

[0015] The corresponding ideal surgical data can be generated from the historical data of the surgical task.

[0016] Ideal surgical data may be generated from historical surgical data associated with positive or ideal outcomes for surgical tasks. If current surgical data deviates from historical surgical data associated with positive outcomes for surgical tasks, it indicates that the surgical outcome of the current surgical procedure may not be as successful / positive as the outcome associated with the historical surgical tasks. By responding to a triggering event by changing the level of automation (which change may include changing whether the current task and / or subsequent tasks are automated), the chance of obtaining a successful surgical outcome may be improved.

[0017] The surgical procedure may include excising tissue. The plurality of surgical tasks may include one or more of: controlling energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, reloading alignment slot management.

[0018] The surgical device may include a memory that stores a plurality of surgical tasks associated with a surgical procedure and the surgical tasks to be automated for each autonomy level.

[0019] The surgical instrument may be adapted to perform two or more surgical procedures, the surgical instrument performing a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure. The surgical instrument may include a memory storing a first plurality of surgical tasks associated with the first surgical procedure and a second plurality of surgical tasks associated with the second surgical procedure. The memory may also store an association between surgical tasks to be automated for each autonomy level for each surgical task in the first plurality of surgical tasks and the second plurality of surgical tasks.

[0020] The surgical instrument may be used in a plurality of procedures, each procedure comprising a different plurality of surgical tasks, and the apparatus may be configured to be able to vary the level of automation of the instrument used to perform any of these procedures.

[0021] The device may be a surgical hub.

[0022] A computer-implemented method for providing commands or suggestions to change the automation level of a surgical instrument used to perform a surgical procedure is described. The method includes receiving an indication of a surgical procedure to be performed using a surgical instrument, wherein the surgical procedure includes a plurality of surgical tasks to be performed using the surgical instrument. The method includes monitoring the performance of a surgical task in the plurality of surgical tasks while the surgical instrument is operating at a first automation level in the plurality of automation levels. The method also includes detecting a trigger event associated with the performance of the surgical task. The method includes providing commands to the surgical instrument or providing suggestions to a user to switch the operation of the surgical instrument from a first automation level to a second automation level based on the trigger event. Advantageously, the computer-implemented method may monitor the performance of the surgical tasks performed by the instrument during the surgical procedure, and provide commands to the instrument or provide suggestions to the user based on the trigger event to change the automation level of the surgical instrument used to perform the surgical procedure, so that the automation level can be increased or decreased to reduce the risk of an adverse surgical outcome of the procedure, and thereby ultimately improve the surgical outcome of the surgical procedure performed. For example, the outcome of tissue resection may correspond to the integrity of the seal line, wherein less seal leakage is equivalent to a more positive outcome than more seal leakage. The outcome of the procedure also corresponds to the number of difficulties that occurred during the surgical procedure, which may include, for example, instrument failure, seal line leakage, staple line misfiring, etc.

[0023] The level of automation may correspond to the number of automated surgical tasks among the plurality of surgical tasks.

[0024] The second automation level may be associated with automating fewer surgical tasks when compared to the first automation level.

[0025] Monitoring the performance of surgical tasks may be based on real-time surgical data. By using real-time surgical data to monitor the performance of surgical tasks, trigger events may be detected and responded to in real time, making it possible to change the level of automation during the current surgical procedure, such as by changing whether the monitored surgical task is automated, and / or changing whether one or more of the subsequent surgical tasks of the surgical procedure are automated, which may increase the chances of a positive surgical outcome for the surgical procedure.

[0026] Real-time surgical data may include at least one of: user data, surgical environment data, surgical instrument data, and task data. For example, surgical instrument data may include physical conditions measured from a surgical device performing a surgical task. User data may include measured physiological conditions of a patient, surgeon, and / or personnel involved in a surgical procedure.

[0027] Monitoring the performance of the surgical task may include comparing one or more of the real-time surgical data with corresponding ideal surgical data, and detecting the occurrence of a triggering event when the comparison exceeds a threshold.

[0028] The corresponding ideal surgical data may be generated from the historical data of the surgical task. The ideal surgical data may be generated from the historical surgical data associated with a positive outcome or an ideal outcome of the surgical task. If the current surgical data deviates from the historical surgical data associated with a positive outcome of the surgical task, it indicates that the surgical outcome of the current surgical operation may not be as successful / positive as the outcome associated with the historical surgical task. By responding to the triggering event by changing the automation level (which change may include changing whether the current task and / or subsequent tasks are automated), the chance of obtaining a successful surgical outcome can be improved.

[0029] The method may further include generating corresponding ideal surgical data.

[0030] The method may include retrieving from a memory a plurality of surgical tasks associated with the surgical procedure and the surgical tasks to be automated for each autonomy level.

[0031] The surgical instrument may be adapted to perform two or more surgical procedures, the surgical instrument performing a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure. The method may also include retrieving from a memory an indication of the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and retrieving from the memory an association between the surgical tasks to be automated for each autonomy level of each surgical task in the first plurality of surgical tasks and the second plurality of surgical tasks.

[0032] The surgical instrument may be used in a plurality of procedures, each procedure comprising a different plurality of surgical tasks, and the apparatus may be configured to be able to vary the level of automation of the instrument used to perform any of these procedures.

[0033] A computer program comprises instructions which, when a computer executes the program, cause the computer to perform any of the previously mentioned methods.

[0034] A computer readable medium comprises instructions which, when executed by a computer, cause the computer to perform any of the previously mentioned methods.

[0035] An apparatus for performing multiple surgical tasks in a surgical procedure is described. The apparatus includes a processor. The processor is configured to be capable of operating at a first automation level. The processor is configured to be capable of obtaining an indication to switch to a second automation level; wherein the indication is based on detecting a trigger condition. The processor is further configured to be capable of operating at the second automation level based on the indication. Advantageously, the apparatus capable of operating at different automation levels allows for changes in automation, enabling a procedure to be more or less automated to reduce the risk of an adverse surgical outcome of the procedure, and thereby ultimately improve the surgical outcome of the surgical procedure performed.

[0036] The first automation level and the second automation level may differ in the number of surgical tasks that are automated during the surgical procedure.

[0037] The indication may be based on monitoring performance of a surgical task among a plurality of surgical tasks at a first level of automation.

[0038] Execution of surgical tasks at a first level of automation may be based on real-time surgical data.

[0039] By using real-time surgical data to monitor the performance of surgical tasks, trigger events can be detected and responded to in real time, making it possible to change the level of automation during the current surgical operation, for example by changing whether the monitored surgical task is automated and / or changing whether one or more of the subsequent surgical tasks of the surgical operation are automated, which can increase the chances of a positive surgical outcome for the surgical operation.

[0040] A surgical procedure may be automated. A device may receive an indication of a surgical task to be performed using a surgical instrument. The capability of the surgical instrument may be associated with an automation level. The device may monitor the performance of the surgical task when the surgical instrument is operated at a first automation level associated with the automation level. The device may detect a trigger event associated with the performance of the surgical task, and, for example, switch the operation of the surgical instrument from the first automation level to a second automation level associated with the automation level based on the trigger event. The capability of the surgical instrument may include a set of surgical instrument tasks, and the automation level may be associated with automating one or more surgical instrument tasks in the set of surgical instrument tasks. The performance of the surgical task may be based on real-time surgical data. The real-time surgical data may include one or more of the following: user data, surgical environment data, surgical instrument data, task data, historical data, etc. In an example, the device may detect a trigger event by comparing the performance of the surgical task with a trigger event threshold, and may switch the operation of the surgical instrument from the first automation level to the second automation level based on the trigger event. When compared to the first automation level, the second automation level may be associated with automating fewer surgical instrument tasks. Monitoring the performance of the surgical task may include comparing one or more of the real-time surgical data with corresponding ideal surgical data.

[0041] In an example, the device may operate at a first automation level in the automation levels associated with the surgical task. The device may obtain an indication to switch to a second automation level in the automation levels associated with the surgical task. The indication may be based on detecting a trigger condition. The device may operate at the second automation level in the automation levels associated with the surgical task based on the indication. The indication may be based on monitoring execution of the first automation level. Execution of the first automation level may be based on real-time surgical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a block diagram of a computer-implemented surgical system.

[0043] Figure 2 An exemplary surgical system is shown in a surgical operating room.

[0044] Figure 3 Exemplary surgical hubs that pair with various systems are illustrated.

[0045] Figure 4 A surgical data network having a set of communicating surgical hubs configured to be able to connect to a set of sensing systems, an environmental sensing system, a set of devices, etc. is illustrated.

[0046] Figure 5 A logic diagram of a control system for a surgical instrument is illustrated.

[0047] Figure 6 An exemplary surgical system is shown that includes a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter.

[0048] Figure 7 An exemplary situational awareness surgical system is shown.

[0049] Figure 8 An example of a surgical autonomous system is shown.

[0050] Fig. 9 Examples of autonomous modules associated with surgical instruments and surgical instrument capabilities are shown.

[0051] Fig.10 An example of a computer-implemented autonomous surgical system is shown.

[0052] Fig.11 An example of monitoring the performance of a surgical task associated with an autonomy level is shown.

[0053] Fig.12 An example of the relationship between error magnitude and autonomy level is shown.

[0054] Fig.13 An example of the relationship between error magnitude and autonomy level is shown.

[0055] Fig.14 An example of the relationship between autonomy level, machine learning, and surgical tasks is shown.

[0056] Fig.15 An example of determining the level of autonomy of a surgical instrument in performing a surgical task is shown.

[0057] Fig.16 An example flow chart for automating surgical tasks associated with levels of autonomy is shown. DETAILED DESCRIPTION

[0058] Figure 1 20000. An exemplary surgical system such as surgical system 20000 may include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, surgical system 20002 may include a computer-implemented interactive surgical system. For example, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 that communicates with a cloud computing system 20008, such as Figure 2The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Exemplary surgical systems 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human-machine interface system 20012, and the like. The human-machine interface system is also referred to herein as a human-machine interface device. The wearable sensing system 20011 may include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include, for example, one or more devices for measuring one or more environmental attributes, such as Figure 2 As further described. The robotic system 20013 may include a plurality of devices for performing a surgical procedure, such as Figure 2 Further described.

[0059] The surgical system 20002 can communicate with a remote server 20009, which can be part of a cloud computing system 20008. In one example, the surgical system 20002 can communicate with the remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, the patient sensing system can communicate directly with the remote server 20009. The surgical system 20002 and / or components therein can communicate with the remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, Advanced LTE (LTE-A), New Radio (NR), or 5G.

[0060] The surgical hub 20006 can cooperate and interact with one of the multiple devices that display images from the laparoscope and information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from one or more sensing systems 20011 and send notification or control messages to the one or more sensing systems 20011. The surgical hub 20006 can send information including notification information to the human-machine interface system 20012 and / or receive information including notification information from the human-machine interface system. The human-machine interface system 20012 may include one or more human-machine interface devices (HIDs). The surgical hub 20006 can send and / or receive notification information or control information to convert into audio, display and / or control information to various devices communicating with the surgical hub.

[0061] For example, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, such as Figure 1 The one or more sensing systems 20001 can measure data related to various biomarkers. The one or more sensing systems 20001 can use one or more sensors such as light sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, pyroelectric sensors, infrared sensors, etc. to measure biomarkers. The one or more sensors can use one or more of the following sensing technologies to measure biomarkers as described herein: photoplethysmography, electrocardiography, electroencephalography, colorimetry, impedance tracing, potentiometric measurement, amperometric measurement, etc.

[0062] Biomarkers measured by the one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximum oxygen uptake, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood pH, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, coughing and sneezing, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, psychiatric factors, sweat, circulating tumor cells, autonomic nervous tone, circadian rhythm and / or menstrual cycle.

[0063] Biomarkers may be related to physiological systems, which may include, but are not limited to, behavior and psychology, cardiovascular system, renal system, skin system, nervous system, gastrointestinal system, respiratory system, endocrine system, immune system, tumor, musculoskeletal system and / or reproductive system. Information from biomarkers can be determined and / or used by, for example, a computer-implemented patient and surgical system 20000. Information from biomarkers can be determined and / or used by a computer-implemented patient and surgical system 20000, for example, to improve the system and / or improve patient outcomes. One or more sensing systems 20001, biomarkers 20005 and physiological systems are described in more detail in U.S. application 17 / 156,287 (Agent File No. END9290USNP1), filed on January 22, 2021, entitled "METHODOF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS", the disclosure of which is incorporated herein by reference in its entirety.

[0064] Figure 2 An example of a surgical system 20002 in a surgical operating room is shown. Figure 2As shown, a patient undergoes surgery by one or more healthcare professionals (HCPs). The HCP is monitored by one or more HCP sensing systems 20020 worn by the HCP. The HCP and the environment around the HCP may also be monitored by one or more environmental sensing systems, which may include, for example, a set of cameras 20021, a set of microphones 20022, and other sensors that may be deployed in an operating room. The HCP sensing system 20020 and the environmental sensing system may communicate with a surgical hub 20006, which in turn may communicate with one or more cloud servers 20009 of a cloud computing system 20008, such as Figure 1 The environmental sensing system may be used to measure one or more environmental attributes, such as the position of the HCP in the operating room, HCP movement, ambient noise in the operating room, temperature / humidity in the operating room, etc.

[0065] like Figure 2 As shown, the main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned in the sterile area to be visible to the operator at the operating table 20024. In addition, the visualization / notification tower 20026 is positioned outside the sterile area. The visualization / notification tower 20026 may include a first non-sterile human-machine interaction device (HID) 20027 and a second non-sterile HID 20029 that are separated from each other. The HID can be a display or a display with a touch screen that allows people to directly interface with the HID. The human-machine interface system guided by the surgical hub 20006 can be configured to be able to coordinate the information flow of operators inside and outside the sterile area using HIDs 20027, 20029, and 20023. In one example, the surgical hub 20006 can cause the HID (e.g., the main HID 20023) to display notifications and / or information about patients and / or surgical procedures. In one example, the surgical hub 20006 can prompt for input and / or receive input from personnel in the sterile or non-sterile field. In one example, the surgical hub 20006 can cause the HID to display a snapshot of the surgical site recorded by the imaging device 20030 on the non-sterile HID 20027 or 20029 while maintaining a real-time feed of the surgical site on the main HID 20023. For example, the snapshot on the non-sterile display 20027 or 20029 can allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0066] In one aspect, the surgical hub 20006 can be configured to be able to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to the primary display 20023 within the sterile field, where the sterile operator at the operating table can view the diagnostic input or feedback. In one example, the input can be a modified form of a snapshot displayed on the non-sterile display 20027 or 20029, which can be routed to the primary display 20023 via the surgical hub 20006.

[0067] refer to Figure 2 , surgical instrument 20031 is used in a surgical procedure as part of surgical system 20002. Hub 20006 can be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, described in U.S. Patent Application Publication No. US2019-0200844A1 (U.S. Patent Application No. 16 / 209,385), filed on December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at visualization tower 20026 can be routed by hub 20006 to a surgical instrument display within the sterile field, where the operator of surgical instrument 20031 can view the diagnostic input or feedback. For example, exemplary surgical instruments suitable for use with surgical system 20002 are described under the heading “Surgical Instrument Hardware” in U.S. patent application publication No. US 2019-0200844A1 (U.S. patent application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0068] Figure 2An example of a surgical system 20002 for performing a surgical procedure on a patient lying flat on an operating table 20024 in a surgical operating room 20035 is shown. A robotic system 20034 may be used in a surgical procedure as part of the surgical system 20002. The robotic system 20034 may include a surgeon's console 20036, a patient side cart 20032 (surgical robot), and a surgical robotic hub 20033. The patient side cart 20032 may manipulate at least one removably coupled surgical tool 20037 through a minimally invasive incision in the patient's body while the surgeon observes the surgical site through the surgeon's console 20036. Images of the surgical site may be obtained through a medical imaging device 20030, which may be manipulated by the patient side cart 20032 to orient the imaging device 20030. The robotic hub 20033 may be used to process images of the surgical site for subsequent display to the surgeon through the surgeon's console 20036.

[0069] Other types of robotic systems may be readily adapted for use with the surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application No. US2019-0201137A1, filed on December 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL” (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0070] Various examples of cloud-based analytics performed by cloud computing system 20008 and suitable for use with the present disclosure are described in U.S. patent application publication No. US2019-0206569 A1 (U.S. patent application No. 16 / 209,403), entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0071] In various aspects, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0072] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0073] One or more illumination sources may be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescent spectrum) is that portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye) and may be referred to as visible light or simply light. A typical human eye will respond to wavelengths in the range of about 380 nm to about 750 nm in air.

[0074] The invisible spectrum (e.g., non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0075] In various aspects, the imaging device 20030 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophago-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0076] The imaging device can use multispectral monitoring to distinguish the topography and the underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. US2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385) filed on December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", the disclosure of which is incorporated herein by reference in its entirety. After completing the surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for relocating the surgical site. It is self-evident that during any surgery, the operating room and surgical equipment need to be strictly sterilized. The strict sanitation and sterilization conditions required in the "surgical room" (i.e., operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including the imaging device 20030 and its attachments and components. It should be understood that the sterile field can be considered as a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered as the area around the patient prepared for surgery. The sterile field can include properly dressed scrubbed team members, as well as all equipment and fixtures in the area.

[0077] Figure 1 The wearable sensing system 20011 shown may include one or more sensing systems, such as Figure 2HCP sensing system 20020 shown. HCP sensing system 20020 may include a sensing system for monitoring and detecting a set of physical states and / or a set of physiological states of a healthcare provider (HCP). The HCP may typically be a surgeon or one or more healthcare providers assisting the surgeon or other healthcare providers. In one example, sensing system 20020 may measure a set of biomarkers to monitor the heart rate of the HCP. In one example, a sensing system 20020 (e.g., a watch or wristband) worn on the surgeon's wrist may use an accelerometer to detect hand motion and / or shaking and determine the amplitude and frequency of the tremor. Sensing system 20020 may send measurement data associated with the set of biomarkers and data associated with the surgeon's physical state to surgical hub 20006 for further processing. One or more environmental sensing devices may send environmental information to surgical hub 20006. For example, an environmental sensing device may include a camera 20021 for detecting the hand / body position of the HCP. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include devices such as thermometers for measuring temperature and hygrometers for measuring humidity of the environment in the operating room. A surgical hub 20006, either alone or in communication with a cloud computing system, may use surgeon biomarker measurement data and / or environmental sensing information to modify the average latency of a control algorithm or a robotic interface of a handheld instrument, for example, to minimize tremor. In one example, the HCP sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and send measurement data associated with the surgeon biomarkers to the surgical hub 20006. The HCP sensing system 20020 may communicate with the surgical hub 20006 using one or more of the following RF protocols: Bluetooth, Bluetooth Low Energy (BLE), Smart Bluetooth, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Wi-Fi. Surgeon biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels associated with surgeons or patients, surgeon and / or personnel movement, surgeon and / or personnel attention levels, etc.

[0078] The surgical hub 20006 can use the surgeon biomarker measurement data associated with the HCP to adaptively control one or more surgical instruments 20031. For example, the surgical hub 20006 can send a control program to the surgical instrument 20031 to control its actuator to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 can send the control program based on situational awareness and / or context about the importance or criticality of the task. When control is needed, the control program can instruct the instrument to change operation to provide more control.

[0079] Figure 3 An exemplary surgical system 20002 with a surgical hub 20006 is shown. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human-machine interface system 20012, a robotic system 20013, and a smart device 20014 via modular controls. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain aspects, as Figure 3 As shown, the hub 20006 also includes a smoke evacuation module 20054 and / or a suction / irrigation module 20055. Various modules and systems can be connected to the modular controls directly via a router or via a communication module 20056. The operating room devices can be coupled to cloud computing resources and data storage devices via the modular controls. The human-machine interface system 20012 may include a display subsystem and a notification subsystem.

[0080] The modular control may be coupled to a non-contact sensor module. The non-contact sensor module may use an ultrasonic, laser-type and / or similar non-contact measuring device to measure the size of the operating room and generate a mapping of the surgical room. Other distance sensors may be used to determine the boundaries of the operating room. The ultrasonic-based non-contact sensor module may scan the operating room by emitting a burst of ultrasound and receiving an echo when it bounces back from the walls of the operating room, as described in the title "Surgical Hub Spatial Awareness Within an Operating Room" in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled "INTERACTIVE SURGICAL PLATFORM" filed on December 28, 2017, which is incorporated herein by reference in its entirety. The sensor module may be configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module may scan the operating room by emitting a laser pulse, receiving a laser pulse that bounces back from the walls of the operating room, and comparing the phase of the emitted pulse with the received pulse to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0081] During surgery, the application of energy to tissue for sealing and / or cutting is often associated with smoke evacuation, suction of excess fluid, and / or flushing of tissue. Fluid lines, power lines, and / or data lines from different sources are often tangled during surgery. Solving this problem during surgery can waste valuable time. Disconnecting a line may require disconnecting the line from its corresponding module, which may require resetting the module. The hub modular housing 20060 provides a unified environment for managing power lines, data lines, and fluid lines, which reduces the frequency of tangling between such lines. Various aspects of the present disclosure provide a surgical hub 20006 for use in surgery involving the application of energy to tissue at a surgical site. The surgical hub 20006 includes a hub housing 20060 and a combined generator module slidably accommodated in a docking base of the hub housing 20060. The docking base includes a data contact and a power contact. The combined generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component seated in a single unit. In one aspect, the combined generator module also includes a smoke exhaust component for connecting the combined generator module to at least one energy delivery cable of a surgical instrument, at least one smoke exhaust component configured to be able to exhaust smoke, fluid and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke exhaust component. In one aspect, the fluid line can be a first fluid line, and the second fluid line can extend from the remote surgical site to the suction and flushing module 20055 slidably accommodated in the hub housing 20060. In one aspect, the hub housing 20060 may include a fluid interface. Certain surgical procedures may require more than one energy type to be applied to the tissue. One energy type may be more conducive to cutting tissue, while another different energy type may be more conducive to sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of the present disclosure provide a solution in which a hub modular housing 20060 is configured to accommodate different generators and facilitate interactive communication between them. One of the advantages of the hub modular housing 20060 is that it enables rapid removal and / or replacement of various modules. Aspects of the present disclosure provide a modular surgical housing for use in surgical procedures involving application of energy to tissue. The modular surgical housing includes a first energy generator module and a first docking base, the first energy generator module being configured to generate a first energy for application to tissue, the first docking base including a first docking port, the first docking port including a first data and power contact, wherein the first energy generator module is slidably movable into electrical engagement with the power and data contact, and wherein the first energy generator module is slidably movable out of electrical engagement with the first power and data contact.Further describing the above, the modular surgical shell also includes a second energy generator module and a second docking base, the second energy generator module is configured to generate a second energy different from the first energy for application to tissue, the second docking base includes a second docking port, the second docking port includes a second data and power contact, wherein the second energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the second energy generator is slidably movable out of electrical contact with the second power and data contacts. In addition, the modular surgical shell also includes a communication bus between the first docking port and the second docking port, which is configured to facilitate communication between the first energy generator module and the second energy generator module. Reference. Figure 3 , aspects of the present disclosure are presented as a hub modular housing 20060 that allows modular integration of a generator module 20050, a smoke evacuation module 20054, and a suction / irrigation module 20055. The hub modular housing 20060 also facilitates interactive communication between the modules 20059, 20054, 20055. The generator module 20050 may have integrated monopolar components, bipolar components, and ultrasonic components supported in a single housing unit that is slidably inserted into the hub modular housing 20060. The generator module 20050 may be configured to be connectable to a monopolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 20060. The hub modular housing 20060 can be configured to facilitate the insertion of multiple generators and interactive communication between the generators docked into the hub modular housing 20060 so that the generators will function as a single generator.

[0082] Figure 4 A surgical data network having a set of communication hubs according to at least one aspect of the present disclosure is shown, the communication hubs being configured to connect a set of sensing systems, environmental sensing systems, and a set of other modular devices located in one or more operating rooms, patient recovery rooms, or rooms in the medical facility specifically equipped for surgical operations to a cloud.

[0083] like Figure 4As shown, the surgical hub system 20060 may include a modular communication hub 20065 that is configured to connect modular devices located in a medical facility to a cloud-based system (e.g., a cloud computing system 20064, which may include a remote server 20067 coupled to a remote storage device 20068). The modular communication hub 20065 and the devices may be connected in a room in the medical facility that is specifically equipped for surgical operations. In one aspect, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicates with a network router 20066. The modular communication hub 20065 may be coupled to a local computer system 20063 to provide local computer processing and data manipulation.

[0084] The computer system 20063 may include a processor and a network interface 20100. The processor may be coupled to a communication module, a storage device, a memory, a non-volatile memory, and an input / output (I / O) interface via a system bus. The system bus may be any of several types of bus structures including a memory bus or a memory controller, a peripheral bus or an external bus, and / or a local bus using any of a variety of available bus architectures, including but not limited to a 9-bit bus, an Industry Standard Architecture (ISA), a Micro Charmel Architecture (MSA), an Extended ISA (EISA), Intelligent Drive Electronics (IDE), a VESA Local Bus (VLB), a Peripheral Component Interconnect (PCI), a USB, an Advanced Graphics Port (AGP), a Personal Computer Memory Card International Association bus (PCMCIA), a Small Computer System Interface (SCSI), or any other peripheral bus.

[0085] The controller may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be a commercially available LM4F230H5QR ARM Cortex-M4F processor core, such as Texas Instruments, which includes 256KB of single-cycle flash or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improved execution above 40MHz, 32KB of single-cycle sequential random access memory (SRAM), a 32KB MCU loaded with Internal read-only memory (ROM) for software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which can be found in the product data sheet.

[0086] In one example, the processor may include a safety controller including two families based on controllers such as TMS570 and RM4x, also known by Texas Instruments under the trade name Hercules ARMCortex R4. The safety controller may be configured to be specifically designed for IEC 61508 and ISO 26262 safety critical applications, etc., to provide advanced integrated safety features while delivering scalable execution, connectivity and memory options.

[0087] It should be understood that the computer system 20063 may include software that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment. Such software may include an operating system. An operating system that may be stored on a disk storage device may be used to control and allocate the resources of a computer system. System applications may utilize the operating system to manage resources through program modules and program data stored in system memory or disk storage devices. It should be understood that the various components described herein may be implemented using various operating systems or combinations of operating systems.

[0088] The user can input commands or information into the computer system 20063 through an input device coupled to the I / O interface. The input device may include, but is not limited to, a pointing device, such as a mouse, a touch ball, a stylus, a touch pad, a keyboard, a microphone, a joystick, a game pad, a satellite dish, a scanner, a TV tuner card, a digital camera, a digital video camera, a web camera, etc. These and other input devices are connected to the processor 20102 through the system bus via an interface port. The interface port includes, for example, a serial port, a parallel port, a game port, and a USB. The output device uses the same type of port as the input device. Therefore, for example, a USB port can be used to provide input to the computer system 20063 and output information from the computer system 20063 to the output device. An output adapter is provided to illustrate that some output devices, such as monitors, displays, speakers, and printers, may need special adapters in other output devices. The output adapter may include, by way of example, but is not limited to, a video and sound card that provides a connection device between the output device and the system bus. It should be noted that other devices or systems such as remote computers can provide both input capabilities and output capabilities.

[0089] Computer system 20063 can be operated in a networked environment using a logical connection with one or more remote computers (such as cloud computers) or local computers. A remote cloud computer can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device or other public network node, etc., and generally includes many or all of the elements described relative to the computer system. For simplicity, only a memory storage device with a remote computer is shown. A remote computer can be logically connected to a computer system via a network interface, and then physically connected via a communication connection. A network interface can cover communication networks such as a local area network (LAN) and a wide area network (WAN). LAN technology can include a fiber distributed data interface (FDDI), a copper distributed data interface (CDDI), Ethernet / IEEE 802.3, a token ring / IEEE 802.5, etc. WAN technology can include, but is not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variants thereof, packet switching networks, and digital subscriber lines (DSL).

[0090] In various examples, computer system 20063 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) for processing digital images. The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instructions, multiple data (MIMD) techniques to improve speed and efficiency. The digital image processing engine may perform a range of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0091] The communication connection may refer to the hardware / software used to connect the network interface to the bus. Although the communication connection is shown for exemplary clarification within the computer system 20063, it may also be located outside the computer system 20063. The hardware / software necessary to connect to the network interface may include internal and external technologies for exemplary purposes only, such as modems, including conventional telephone-grade modems, cable modems, fiber optic modems and DSL modems, ISDN adapters, and Ethernet cards. In some examples, an RF interface may also be used to provide a network interface.

[0092] The surgical data network associated with the surgical hub system 20060 can be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, enabling it to be transferred from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network includes additional features to enable monitoring of traffic through the surgical data network and configuration of each port in the network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0093] Modular devices 1a-1n located in an operating room may be coupled to a modular communication hub 20065. A network hub 20061 and / or a network switch 20062 may be coupled to a network router 20066 to connect devices 1a-1n to a cloud computing system 20064 or a local computer system 20063. Data associated with devices 1a-1n may be transmitted to a cloud-based computer via a router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transmitted to a local computer system 20063 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be coupled to a network switch 20062. A network switch 20062 may be coupled to a network hub 20061 and / or a network router 20066 to connect devices 2a-2m to a cloud 20064. Data associated with devices 2a-2m may be transmitted to a cloud computing system 20064 via a network router 20066 for data processing and manipulation. Data associated with the devices 2a-2m may also be transmitted to the local computer system 20063 for local data processing and manipulation.

[0094] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing system 20069 may include an HCP sensing system and / or a patient sensing system. The one or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly via one of the network routers 20066 or via a network hub 20061 or a network switch 20062 that communicates with the network router 20066.

[0095] The sensing system 20069 may be coupled to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 may be transmitted to the cloud computing system 20064 for data processing and manipulation via the network router 20066. Data associated with the sensing system 20069 may also be transmitted to the local computer system 20063 for local data processing and manipulation.

[0096] like Figure 4 As shown, the surgical hub system 20060 can be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. A modular communication hub 20065 can be included in a modular control tower that is configured to accommodate multiple devices 1a-1n / 2a-2m. A local computer system 20063 can also be included in the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images obtained by some of the devices 1a-1n / 2a-2m during a surgical procedure, for example. In various aspects, the devices 1a-1n / 2a-2m can include, for example, various modules, such as an imaging module coupled to an endoscope, a generator module coupled to an energy-based surgical device, a smoke evacuation module, a suction / irrigation module, a communication module, a processor module, a storage array, a surgical device connected to a display, and / or a non-contact sensor module in other modular devices of the modular communication hub 20065 that can be connected to a surgical data network.

[0097] In one aspect, Figure 4The surgical hub system 20060 shown may include a combination of a network hub, a network switch, and a network router that connects the device 1a-1n / 2a-2m or the sensing system 20069 to the cloud base system 20064. One or more of the devices 1a-1n / 2a-2m or the sensing system 20069 coupled to the network hub 20061 or the network switch 20062 can collect data in real time and transmit the data to the cloud computer for data processing and operation. It should be understood that cloud computing relies on shared computing resources rather than using local servers or personal devices to process software applications. The term "cloud" can be used as a metaphor for the "Internet", although the term is not limited to this. Thus, the term "cloud computing" may be used herein to refer to a "type of Internet-based computing" in which different services (such as servers, storage devices, and applications) are delivered to a modular communication hub 20065 and / or computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space) and devices connected to the modular communication hub 20065 and / or computer system 20063 via the Internet. The cloud infrastructure may be maintained by a cloud service provider. In this case, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services can perform a large number of calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to be connected to a computer that communicates with cloud computing resources and storage devices.

[0098] By applying cloud computer data processing technology to the data collected by the devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical results, reduced costs and improved patient satisfaction. At least some of the devices 1a-1n / 2a-2m can be used to observe tissue status to evaluate the leakage or perfusion of the sealed tissue after tissue sealing and cutting surgery. At least some of the devices 1a-1n / 2a-2m can be used to identify pathology, such as the effects of disease, using cloud-based computing to examine data including images of body tissue samples for diagnostic purposes. This may include positioning and edge confirmation of tissue and phenotype. At least some of the devices 1a-1n / 2a-2m can be used to identify the anatomical structure of the body using various sensors integrated with imaging devices and technologies (such as overlapping images captured by multiple imaging devices). The data (including image data) collected by the devices 1a-1n / 2a-2m can be transmitted to the cloud computing system 20064 or the local computer system 20063 or both for data processing and manipulation, including image processing and manipulation. The data can be analyzed to improve surgical outcomes by determining whether further treatment can be pursued, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions. Such data analysis can be further processed using outcome analysis, and using standardized methods can provide helpful feedback to confirm surgical treatments and surgeon behavior, or suggest modifications to surgical treatments and surgeon behavior.

[0099] Applying cloud computer data processing techniques to the measurement data collected by the sensing systems 20069, the surgical data network can provide improved surgical results, improved recovery results, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 can be used to assess the physiological condition of a surgeon operating on a patient, or a patient preparing for surgery, or a patient recovering after surgery. The cloud-based computing system 20064 can be used to monitor biomarkers associated with the surgeon or patient in real time, and can be used to generate a surgical plan based at least on the measurement data collected prior to surgery, provide control signals to surgical instruments during surgery, and notify patients of complications during the post-operative period.

[0100] The operating room devices 1a-1n can be connected to the modular communication hub 20065 through a wired channel or a wireless channel, depending on the configuration of the devices 1a-1n to the network hub 20061. In one aspect, the network hub 20061 can be implemented as a local network broadcast device that works on the physical layer of the open system interconnection (OSI) model. The network hub can provide connections with devices 1a-1n located in the same operating room network. The network hub 20061 can collect data in the form of packets and send it to the router in half-duplex mode. The network hub 20061 may not store any media access control / Internet protocol (MAC / IP) for transmitting device data. Only one device in the device 1a-1n can send data through the network hub 20061 at a time. The network hub 20061 may not have a routing table or intelligence about where to send information and broadcast all network data on each connection and broadcast all network data to the remote server 20067 of the cloud computing system 20064. Network hub 20061 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose a security risk and cause bottlenecks.

[0101] The operating room devices 2a-2m may be connected to the network switch 20062 via a wired channel or a wireless channel. The network switch 20062 operates in the data link layer of the OSI model. The network switch 20062 may be a multicast device for connecting the devices 2a-2m located in the same operating room to the network. The network switch 20062 may send data to the network router 20066 in the form of a frame and may operate in full-duplex mode. Multiple devices 2a-2m may simultaneously send data via the network switch 20062. The network switch 20062 stores and uses the MAC addresses of the devices 2a-2m to transmit data.

[0102] The network hub 20061 and / or the network switch 20062 may be coupled to a network router 20066 to connect to the cloud computing system 20064. The network router 20066 operates in the network layer of the OSI model. The network router 20066 generates routes for transmitting data packets received from the network hub 20061 and / or the network switch 20062 to cloud-based computer resources to further process and manipulate data collected by any or all of the devices 1a-1n / 2a-2m and the wearable sensing system 20011. The network router 20066 may be employed to connect two or more different networks located in different locations, such as, for example, different operating rooms in the same medical facility or different networks located in different operating rooms in different medical facilities. The network router 20066 may send data to the cloud computing system 20064 in the form of packets and operate in full-duplex mode. Multiple devices may send data simultaneously. The network router 20066 may use IP addresses to transmit data.

[0103] In one example, the network hub 20061 may be implemented as a USB hub that allows multiple USB devices to be connected to a host computer. A USB hub may expand a single USB port to multiple levels so that more ports are available to connect devices to a host system computer. The network hub 20061 may include wired or wireless capabilities for receiving information through a wired channel or a wireless channel. In one aspect, a wireless USB short-range, high-bandwidth wireless radio communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in an operating room.

[0104] In an example, the operating room devices 1a-1n / 2a-2m and / or the sensing system 20069 can communicate with the modular communication hub 20065 via the Bluetooth wireless technology standard for exchanging data from fixed devices and mobile devices over short distances (using short wavelength UHF radio waves from 2.4GHz to 2.485GHz in the ISM band) and building a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or the sensing system 20069 can communicate with the modular communication hub 20065 via a variety of wireless or wired communication standards or protocols, including but not limited to Bluetooth, Low-Energy Bluetooth, Near Field Communication (NFC), Wi-Fi (IEEE802.11 series), WiMAX (IEEE 802.16 series), IEEE 802.20, New Radio (NR), Long Term Evolution (LTE) and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT and its Ethernet derivatives, and any other wireless and wired protocols designated as 3G, 4G, 5G and above. The computing module may include multiple communication modules. For example, the first communication module can be dedicated to shorter-range wireless communications, such as Wi-Fi and Bluetooth Low Energy, Smart Bluetooth, while the second communication module can be dedicated to longer-range wireless communications, such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, etc.

[0105] Modular communication hub 20065 may serve as a central connection for one or more of operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 and may process data types referred to as frames. Frames may carry data generated by devices 1a-1n / 2a-2m and / or sensing systems 20069. When modular communication hub 20065 receives a frame, the frame may be amplified and / or sent to a network router 20066, which may transmit the data to a cloud computing system 20064 or a local computer system 20063 using a plurality of wireless or wired communication standards or protocols, as described herein.

[0106] Modular communication hub 20065 can be used as a standalone device or connected to a compatible network hub 20061 and network switch 20062 to form a larger network. Modular communication hub 20065 is generally easy to install, configure, and maintain, making it a good option for networking operating room devices 1a-1n / 2a-2m.

[0107] Figure 5 A logic diagram of a control system 20220 of a surgical instrument or surgical tool according to one or more aspects of the present disclosure is shown. The surgical instrument or surgical tool may be configurable. The surgical instrument may include a surgical fixation device dedicated to the surgery to be performed, such as an imaging device, a surgical stapler, an energy device, an endoscopic cutter device, etc. For example, the surgical instrument may include any of an electric stapler, a powered stapler generator, an energy device, a pre-energy device, a pre-energy jaw device, an endoscopic cutter clamp, an energy device generator, an operating room imaging system, a smoke evacuator, a suction-irrigation device, an insufflation system, etc. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221, which includes a processor 20222 and a memory 20223. For example, one or more of sensors 20225, 20226, 20227 provide real-time feedback to the processor 20222. A motor 20230 driven by a motor driver 20229 may be operably coupled to a longitudinally movable displacement member to drive an I-beam blade element. Tracking system 20228 can be configured to determine the position of the longitudinally movable displacement member. Position information can be provided to processor 20222, which can be programmed or configured to determine the position of the longitudinally movable drive member and the position of the firing member, firing rod and I-beam blade element. Additional motors can be provided at the tool driver interface to control I-beam firing, closure tube travel, shaft rotation and joint movement. Display 20224 can display a variety of operating conditions of the instrument and can include a touch screen function for data input. The information displayed on the display 20224 can be superimposed with an image obtained via an endoscopic imaging module.

[0108] The microcontroller 20221 may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the main microcontroller 20221 may be a LM4F230H5QR ARM Cortex-M4F processor core available from, for example, Texas Instruments, which includes 256KB of single-cycle flash memory or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improving performance above 40MHz, 32KB of single-cycle SRAM, a CPU loaded with Internal ROM for software, 2KB EEPROM, one or more PWM modules, one or more QEI analog and / or one or more 12-bit ADC with 12 analog input channels, details of which can be found in the product datasheet.

[0109] The microcontroller 20221 may include a safety controller including two controller-based families such as TMS570 and RM4x, known also from Texas Instruments under the trade name Hercules ARM CortexR4. The safety controller may be configured to be specifically used for IEC 61508 and ISO 26262 safety-critical applications, etc., to provide advanced integrated safety features while delivering scalable execution, connectivity and memory options.

[0110] The microcontroller 20221 can be programmed to perform various functions, such as precise control of the speed and position of the knife and articulation system. In one aspect, the microcontroller 20221 may include a processor 20222 and a memory 20223. The electric motor 20230 may be a brushed DC (DC) motor having a gearbox and a mechanical link to the articulation or knife system. In one aspect, the motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers can be easily replaced for use in a tracking system 20228 including an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, entitled "SYSTEMS AND METHODS FOR CONTROLLING ASURGICAL STAPLING AND CUTTING INSTRUMENT," published on October 19, 2017, which is incorporated herein by reference in its entirety.

[0111] The microcontroller 20221 can be programmed to provide precise control of the speed and position of the displacement member and articulation system. The microcontroller 20221 can be configured to be able to calculate the response in the software of the microcontroller 20221. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used for actual feedback decisions. The observed response can be a favorable tuning value that balances the smooth continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0112] The motor 20230 may be controlled by a motor driver 20229 and may be employed by a firing system of a surgical instrument or tool. In various forms, the motor 20230 may be a brushed DC drive motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver including a field effect transistor (FET). The motor 20230 may be powered by a power assembly that may be releasably mounted to a handle assembly or a tool housing for supplying control power to a surgical instrument or tool. The power assembly may include a battery that may include a plurality of battery cells connected in series that may be used as a power source to provide power to a surgical instrument or tool. In some cases, the battery cells of the power assembly may be replaceable and / or rechargeable. In at least one example, the battery cell may be a lithium-ion battery that may be coupled to the power assembly and detachable from the power assembly.

[0113] The motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal oxide semiconductor field effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include a unique charge pump regulator that provides full (>10V) gate drive for battery voltages as low as 7V and allows the A3941 to operate at a reduced gate drive as low as 5.5V. A bootstrap capacitor may be used to provide the above-mentioned battery supply voltage required for the N-channel MOSFET. The internal charge pump of the high-side driver may allow DC (100% duty cycle) operation. A diode or synchronous rectification may be used to drive the full bridge in fast decay mode or slow decay mode. In slow decay mode, current recirculation may pass through the high-side FET or the low-side FET. The power FET may be protected from breakdown by a resistor-adjustable dead time. Comprehensive diagnostics provide indications of undervoltage, overtemperature, and power bridge faults, and can be configured to protect the power MOSFETs under most short circuit conditions.Other motor drivers can be easily substituted for use in tracking systems 20228 including absolute positioning systems.

[0114] The tracking system 20228 may include a controlled motor drive circuit arrangement including a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system can provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member can represent a longitudinally movable drive member, which includes a rack of drive teeth for meshing engagement with a corresponding drive gear of the gear reducer assembly. In some examples, the displacement member can represent a firing member, which can be adapted and configured to include a rack of drive teeth. In some examples, the displacement member can represent a firing rod or an I-beam, each of which can be adapted and configured to include a rack of drive teeth. Therefore, as used herein, the term displacement member can be generally used to refer to any movable member of a surgical instrument or tool such as a drive member, a firing member, a firing rod, an I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member can be coupled to the firing member, the firing rod, and the I-beam. Therefore, the absolute positioning system can actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member can be coupled to any position sensor 20225 suitable for measuring linear displacement. Therefore, the longitudinally movable drive member, firing member, firing rod or I-beam or a combination thereof can be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor. The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a sliding potentiometer, a magnetic sensing system including a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system including a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system including a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system including a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0115] The electric motor 20230 may include a rotatable shaft operably docked with a gear assembly that is mounted on a displacement member in meshing engagement with a set of drive teeth or a rack of drive teeth. The sensor element may be operably coupled to the gear assembly so that a single rotation of the position sensor 20225 element corresponds to a certain linear longitudinal translation of the displacement member. The arrangement of the transmission and sensor may be connected to a linear actuator via a rack and pinion arrangement, or to a rotary actuator via a spur gear or other connection. The power source may supply power to the absolute positioning system, and the output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member that includes a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable firing member, a firing rod, an I-beam, or a combination thereof.

[0116] A single rotation of the sensor element associated with the position sensor 20225 may be equivalent to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" following a single rotation of the sensor element coupled to the displacement member. The sensor arrangement may be connected via a gear reduction that enables the position sensor 20225 to complete one or more rotations for a full stroke of the displacement member. The position sensor 20225 may complete multiple rotations for a full stroke of the displacement member.

[0117] A series of switches (where n is an integer greater than one) may be employed, alone or in combination with gear reduction, to provide unique position signals for more than one rotation of the position sensor 20225. The state of the switches may be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is provided to the microcontroller 20221. The position sensor 20225 of this sensor arrangement may include an array of magnetic sensors, analog rotation sensors (such as potentiometers), or analog Hall effect elements, which output a unique combination of position signals or values.

[0118] Position sensor 20225 may include any number of magnetic sensing elements, such as, for example, magnetic sensors classified according to whether they measure the total magnetic field or the vector components of the magnetic field. The technology used to produce the above two types of magnetic sensors can cover many aspects of physics and electronics. The technology used for magnetic field sensing may include probe coils, fluxgates, optical pumps, nuclear spins, superconducting quantum interference devices (SQUIDs), Hall effects, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetosensitive diodes, magnetosensitive transistors, optical fibers, magneto-optics, and magnetic sensors based on microelectromechanical systems, etc.

[0119] The position sensor 20225 for the tracking system 20228 including the absolute positioning system may include a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT monolithic magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 20225 interfaces with the microcontroller 20221 to provide an absolute positioning system. The position sensor 20225 may be a low voltage and low power component and may include four Hall effect elements in the area of ​​the position sensor 20225 that may be located above the magnet. A high resolution ADC and an intelligent power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor (also known as the bitwise method and Volder algorithm) may be provided to perform a simple and efficient algorithm to calculate hyperbolic and trigonometric functions, which requires only addition, subtraction, bit shifting, and table lookup operations. The angular position, alarm bit, and magnetic field information may be transmitted to the microcontroller 20221 via a standard serial communication interface, such as a serial peripheral interface (SPI) interface. The position sensor 20225 may provide 12 or 14 bit resolution. The position sensor 20225 may be an AS5055 chip provided in a small QFN 16-pin 4x4x0.85mm package.

[0120] The tracking system 20228 including an absolute positioning system may include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power source converts the signal from the feedback controller into a physical input to the system: in this case, voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensors may be provided to measure physical parameters of the physical system. In some aspects, one or more other sensors may include sensor arrangements such as those described in U.S. Patent No. 9,345,481, entitled “STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM,” issued on May 24, 2016, which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, entitled “STAPLE CARTRIDGE TISSUETHICKNESS SENSOR SYSTEM,” issued on September 18, 2014, which is incorporated herein by reference in its entirety; and U.S. Patent Application Serial No. 15 / 628,175, entitled “TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT,” filed on June 20, 2017, which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system will have a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuits to combine the calculated response with the measured response using an algorithm (such as weighted average and theoretical control loop) that drives the calculated response toward the measured response. The calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, inductance and resistance to predict the state and output of the physical system by knowing the input.

[0121] Thus, the absolute positioning system can provide the absolute position of the displacement member when the instrument is powered on, and does not retract or advance the displacement member to a reset (zero or home) position as may be required by conventional rotary encoders, which simply count the number of forward or backward steps taken by the motor 20230 to infer the position of the device actuator, drive rod, knife, etc.

[0122] Sensor 20226 (such as, for example, a strain gauge or micro strain gauge) can be configured to measure one or more parameters of the end effector, such as, for example, the amplitude of the strain applied to the anvil during the clamping operation, which can indicate the closing force applied to the anvil. The measured strain can be converted into a digital signal and provided to the processor 20222. Alternatively or in addition to sensor 20226, sensor 20227 (such as, for example, a load sensor) can measure the closing force applied to the anvil by the closed drive system. Sensor 20227 (such as, for example, a load sensor) can measure the firing force applied to the I-beam during the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sliding member, which is configured to enable the nail driver to cam upward to push the nail out to deform and contact the anvil. The I-beam may also include a sharp cutting edge, which can be used to cut tissue when the I-beam is pushed distally by the firing rod. Alternatively, a current sensor 20231 can be employed to measure the current drawn by the motor 20230. For example, the force required to advance the firing member can correspond to the current drawn by the motor 20230. The measured force can be converted into a digital signal and provided to the processor 20222.

[0123] For example, a strain gauge sensor 20226 can be used to measure the force applied to the tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force on the tissue processed by the end effector. A system for measuring the force applied to the tissue grasped by the end effector may include a strain gauge sensor 20226, such as a micro strain gauge, which can be configured to measure, for example, one or more parameters of the end effector. In one aspect, the strain gauge sensor 20226 can measure the amplitude or magnitude of the strain applied to the jaw member of the end effector during the clamping operation, which can indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 20222 of the microcontroller 20221. The load sensor 20227 can measure the force used to operate the knife element, for example, to cut the tissue captured between the anvil and the nail magazine. A magnetic field sensor can be used to measure the thickness of the captured tissue. The measurement results of the magnetic field sensor can also be converted into a digital signal and provided to the processor 20222.

[0124] The microcontroller 20221 can characterize the corresponding values ​​of the selected position of the firing member and / or the velocity of the firing member using the measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue measured by the sensors 20226, 20227, respectively. In one embodiment, the memory 20223 can store techniques, formulas, and / or lookup tables that can be used by the microcontroller 20221 in the evaluation.

[0125] The control system 20220 of the surgical instrument or tool may also include wired or wireless communication circuitry to communicate with the surgical hub 20065, such as Figure 4 shown.

[0126] Figure 6 An exemplary surgical system 20280 according to the present disclosure is shown and may include a surgical instrument 20282 that communicates with a console 20294 or a portable device 20296 via a wired and / or wireless connection through a local area network 20292 and / or a cloud network 20293. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein to measure the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in situ loading or multi-fire loading unit (MFLU) that allows the clinician to fire multiple fasteners multiple times without removing the loading unit 20287 from the surgical site to reload the loading unit 20287.

[0127] The first jaw 20291 and the second jaw 20290 can be configured to clamp tissue therebetween, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 20291 can be configured to fire at least one fastener multiple times, or can be configured to include a replaceable multiple firing fastener bin, which includes multiple fasteners (e.g., nails, clamps, etc.) that can be fired more than once before being replaced. The second jaw 20290 may include an anvil that deforms the fastener or otherwise fixes the fastener when the fastener is ejected from the multiple firing fastener bin.

[0128] The handle 20297 may include a motor coupled to the drive shaft to affect the rotation of the drive shaft. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include a button, a switch, a lever, a slider, a touch screen, and any other suitable input mechanism or user interface that can be engaged by a clinician to activate the motor.

[0129] The control interface of the handle 20297 can communicate with the controller 20298 of the handle 20297 to selectively activate the motor to affect the rotation of the drive shaft. The controller 20298 can be disposed within the handle 20297 and configured to receive input from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 can also include a display that can be viewed by a clinician during use of the handle 20297. The display can be configured to display a portion of the adapter or loading unit data before, during, or after firing the instrument 20282.

[0130] The adapter 20285 may include an adapter identification device 20284 disposed therein, and the loading unit 20287 may include a loading unit identification device 20288 disposed therein. The adapter identification device 20284 may communicate with the controller 20298, and the loading unit identification device 20288 may communicate with the controller 20298. It should be understood that the loading unit identification device 20288 may communicate with the adapter identification device 20284, which relays or transmits communications from the loading unit identification device 20288 to the controller 20298.

[0131] The adapter 20285 may also include a plurality of sensors 20286 (one is shown) disposed about it to detect various conditions of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of firings of the adapter 20285, the peak force of the adapter 20285 during firing, the total amount of force applied to the adapter 20285, the peak retraction force of the adapter 20285, the number of pauses of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to the adapter identification device 20284 in the form of data signals. The data signals of the plurality of sensors 20286 may be stored in the adapter identification device 20284, or may be used to update the adapter data stored in the adapter identification device. The data signals of the plurality of sensors 20286 may be analog or digital. The plurality of sensors 20286 may include a force gauge to measure the force applied to the loading unit 20287 during firing.

[0132] The handle 20297 and the adapter 20285 can be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface can be a direct electrical interface (i.e., comprising electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface can be a contactless electrical interface to wirelessly transmit energy and signals therebetween (e.g., inductive transmission). It is also conceivable that the adapter identification device 20284 and the controller 20298 can communicate wirelessly with each other via a wireless connection separate from the electrical interface.

[0133] The handle 20297 may include a transceiver 20283 that is configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., the LAN 20292, the cloud 20293, the console 20294, or the portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 20280. For example, the controller 20298 may transmit instrument data to the console 20294, the instrument data including the serial number of the attachment adapter (e.g., adapter 20285) attached to the handle 20297, the serial number of the loading unit (e.g., loading unit 20287) attached to the adapter 20285, and the serial number of the multi-fire fastener cartridge loaded into the loading unit. Thereafter, the console 20294 may transmit data associated with the attached cartridge, loading unit, and adapter, respectively (e.g., cartridge data, loading unit data, or adapter data) back to the controller 20298. The controller 20298 may display a message on a local instrument display, or transmit the message to the console 20294 or the portable device 20296 via the transceiver 20283 to display the message on the display 20295 or the portable device screen, respectively.

[0134] Figure 7A diagram of a situational awareness surgical system 5100 is illustrated in accordance with at least one aspect of the present disclosure. Data sources 5126 may include, for example, modular devices 5102 (the modular devices may include sensors configured to detect parameters associated with the patient, HCP, and environment and / or the modular devices themselves), databases 5122 (e.g., an EMR database containing patient records), and patient monitoring devices 5124 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor), HCP monitoring devices 35510, and / or environmental monitoring devices 35512. The surgical hub 5104 may be configured to be able to derive contextual information related to the surgical procedure from the data, for example, based on a particular combination of the received data or a particular order in which the data is received from the data sources 5126. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being the subject of the surgery. This ability of some aspects of the surgical hub 5104 to derive or infer information related to the surgical procedure from the received data may be referred to as "situational awareness." For example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104 that derives contextual information related to the surgical procedure from data received and / or surgical planning information received from the edge computing system 35514 or the enterprise cloud server 35516.

[0135] The situational awareness system of the surgical hub 5104 may be configured to be able to derive contextual information from data received from the data source 5126 in a variety of different ways. For example, the situational awareness system may include a pattern recognition system or a machine learning system (e.g., an artificial neural network) that has been trained on training data to associate various inputs (e.g., data from the database 5122, the patient monitoring device 5124, the modular device 5102, the HCP monitoring device 35510, and / or the environmental monitoring device 35512) with corresponding contextual information about the surgical procedure. The machine learning system may be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In an example, the situational awareness system may include a lookup table that stores pre-characterized environmental information about the surgical procedure associated with one or more inputs (or input ranges) corresponding to environmental information. In response to a query using one or more inputs, the lookup table may return corresponding contextual information used by the situational awareness system to control the modular device 5102. In an example, the contextual information received by the situational awareness system of the surgical hub 5104 may be associated with a specific control adjustment or a set of control adjustments for one or more modular devices 5102. In an example, the situational awareness system may include an additional machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 5102 when contextual information is provided as input.

[0136] The surgical hub 5104 incorporating a situational awareness system can provide numerous benefits to the surgical system 5100. One benefit can include improved interpretation of sensed and collected data, which in turn will improve processing accuracy and / or use of the data during the surgical procedure. Returning to the previous example, the situational awareness surgical hub 5104 can determine the type of tissue being operated on; thus, when an unexpectedly high force is detected for the end effector of a closing surgical instrument, the situational awareness surgical hub 5104 can properly ramp up or ramp down the motor speed of the surgical instrument for the tissue type.

[0137] The type of tissue being operated on can affect the adjustment of the compression rate and load threshold of the surgical stapling and cutting instrument for a specific tissue gap measurement value. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic surgery or an abdominal surgery, thereby allowing the surgical hub 5104 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument for the type of tissue.

[0138] The type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the type of surgery. Since one type of surgery can often be performed in a particular body cavity, the surgical hub 5104 can then control the motor rate of the smoke evacuator appropriately for the body cavity in which the operation is being performed. Thus, the situational awareness surgical hub 5104 can provide consistent smoke evacuation for both thoracic and abdominal surgeries.

[0139] The type of surgery being performed can affect the optimal energy level at which an ultrasonic surgical instrument or radio frequency (RF) electrosurgical instrument operates. For example, arthroscopic surgery may require a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in a fluid. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (e.g., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The situational awareness surgical hub 5104 can determine the type of surgical procedure being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile of the surgical procedure. In addition, the situational awareness surgical hub 5104 can be configured to be able to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the surgical procedure rather than just on a surgery-by-surgery basis. The situational awareness surgical hub 5104 can determine the step of a surgical procedure being performed or to be subsequently performed and then update the control algorithm of the generator and / or ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type based on the surgical step.

[0140] In an example, data may be extracted from additional data sources 5126 to improve conclusions extracted by the surgical hub 5104 from one data source 5126. The situational awareness surgical hub 5104 may augment the data it receives from the modular device 5102 with contextual information about the surgical procedure that has been constructed from other data sources 5126. For example, the situational awareness surgical hub 5104 may be configured to be able to determine whether hemostasis has occurred (e.g., whether bleeding at a surgical site has stopped) based on video or image data received from a medical imaging device. The surgical hub 5104 may be further configured to be able to compare physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device communicatively coupled to the surgical hub 5104) to determine the integrity of the staple line or tissue weld. The situational awareness system of the surgical hub 5104 may take into account the physiological measurement data to provide additional context when analyzing the visual data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0141] For example, if the situational awareness surgical hub 5104 determines that a subsequent step of the surgery requires the use of an RF electrosurgical instrument, it can actively activate the generator connected to the instrument. Actively activating the energy source can allow the instrument to be ready for use as soon as the previous step of the surgery is completed.

[0142] The situational awareness surgical hub 5104 may determine whether the current or subsequent steps of the surgical procedure require a different view or degree of magnification on the display based on the features that the surgeon anticipates needing to view at the surgical site. The surgical hub 5104 may proactively change the displayed view (e.g., provided by a medical imaging device for the visualization system) accordingly, such that the display is automatically adjusted throughout the surgical procedure.

[0143] The situational awareness surgical hub 5104 can determine which step of the surgical procedure is being performed or will be performed next and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the step of the surgical procedure being performed without having to wait for the surgeon to request that specific information.

[0144] Errors may be checked during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 5104 may determine whether the operating room is correctly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 may be configured to be able to determine the type of surgical procedure being performed, retrieve the corresponding list, product location, or setup requirements (e.g., from a memory), and then compare the current operating room layout with the standard layout determined by the surgical hub 5104 for the type of surgical procedure being performed. In some examples, the surgical hub 5104 may compare a list of items for surgery and / or a list of devices paired with the surgical hub 5104 with a suggested or expected list of items and / or devices for a given surgical procedure. If there is any discontinuity between the lists, the surgical hub 5104 may provide an alert indicating that a particular modular device 5102, a patient monitoring device 5124, an HCP monitoring device 35510, an environmental monitoring device 35512, and / or other surgical items are missing. In some examples, the surgical hub 5104 can determine the relative distance or position of the modular device 5102 and the patient monitoring device 5124, for example, via a proximity sensor. The surgical hub 5104 can compare the relative position of the devices to a suggested or expected layout for a particular surgical procedure. If there are any discontinuities between the layouts, the surgical hub 5104 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the suggested layout.

[0145] The situational awareness surgical hub 5104 may determine if a surgeon (or other HCP) is making an error or otherwise deviating from an expected course of action during a surgical procedure. For example, the surgical hub 5104 may be configured to determine the type of surgical procedure being performed, retrieve a corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the surgical procedure to the expected steps or equipment determined by the surgical hub 5104 for the type of surgical procedure being performed. The surgical hub 5104 may provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0146] The surgical instruments (and other modular devices 5102) can be adjusted for the specific context of each surgical procedure (such as adjusting to different tissue types) and verification actions during the surgical procedure. The next steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the surgical room according to the specific context of the surgery.

[0147] Surgical autonomous systems, devices, and methods may include aspects of integration with other medical devices, data sources, processes, and mechanisms. For example, surgical autonomous systems, devices, and methods may include aspects of integration with a computer-implemented interactive surgical system and / or one or more elements of a computer-implemented interactive surgical system. As described herein, surgical system, surgical autonomous system, and autonomous surgical system are interchangeable.

[0148] refer to Figure 8 , which may provide an overview of the surgical autonomous system 47000. Surgical instrument A 47005 and / or surgical instrument B 47010 may be used in surgical procedures as part of the surgical system 47000. The surgical hub 47040 may be configured to coordinate the flow of information to displays of surgical instruments. For example, the surgical hub may be described in U.S. Patent Application Publication No. US2019-0200844A1 (U.S. Patent Application No. 16 / 209,385), filed on December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” the disclosure of which is incorporated herein by reference in its entirety. For example, exemplary surgical instruments suitable for use with surgical system 47000 are described under the heading “Surgical Instrument Hardware” in U.S. patent application publication No. US2019-0200844 A1 (U.S. patent application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0149] Figure 8 An example of a surgical autonomous system 47000 is shown. System 47000 can be used to perform a surgical procedure on a patient lying flat on an operating table in a surgical operating room. The robotic system can be used in a surgical procedure as part of a surgical system. For example, the robotic system may be described in U.S. Patent Application Publication No. US2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. The robotic hub can be used to process images of the surgical site for subsequent display to the surgeon via the surgeon's console.

[0150] Other types of robotic systems may be readily adapted for use with the surgical system 47000. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. patent application Ser. No. US2019-0201137A1, filed Dec. 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL” (U.S. Patent Application Ser. No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0151] Various examples of cloud-based analytics performed by the cloud and suitable for use with the present disclosure are described in U.S. patent application publication No. US2019-0206569 A1, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” filed on December 4, 2018 (U.S. patent application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0152] In various aspects, an imaging device can be used in the surgical system 47000 and can include at least one image sensor and one or more optical components. Suitable image sensors can include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0153] The optical components of the imaging device may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0154] One or more illumination sources may be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescence spectrum) is that portion of the electromagnetic spectrum that is visible to (e.g., detectable by) the human eye and may be referred to as "visible light" or simply "light." A typical human eye will respond to wavelengths in the range of about 380 nm to about 750 nm in air.

[0155] The invisible spectrum (e.g., non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0156] In various aspects, the imaging device can be configured for use in minimally invasive surgery. Examples of imaging devices suitable for the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophago-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0157] The imaging device can use multispectral monitoring to distinguish the topography and the underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. US2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. After completing the surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site. It is self-evident that during any surgery, the operating room and surgical equipment need to be strictly sterilized. The strict sanitation and sterilization conditions required in the "surgical room" (i.e., operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including imaging devices and their attachments and components. It should be understood that the sterile field can be considered as a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered as the area around the patient prepared for surgery. The sterile field can include properly dressed scrubbed team members, as well as all equipment and fixtures in the area.

[0158] As about Figure 2 , Figure 3 , Figure 5 and Figure 7 As described, surgical instrument A 47005 and / or surgical instrument B 47010 may include one or more capabilities (e.g., capability 47015 associated with surgical instrument A 47005, which may include B, Z, and D, and capability 47020 associated with surgical instrument B 470010, which may include C, F, and E). These capabilities may be associated with functions that the surgical instrument is capable of performing. Examples of these functions may be described under the title “Surgical Instrument Hardware” in U.S. patent application publication No. US 2019-0200844A1 (U.S. patent application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. For example, if the surgical instrument is an endoscopic cutter, one of the capabilities may be the removal of tissue (e.g., tissue surrounding the colon when a physician is performing a proctocolectomy). These capabilities may include, for example, Fig. 9 The surgical tasks described. For example, the ability to remove tissue may include controlling energy source, cutting, suturing, knob orientation, body orientation, body position, anvil jaw force, reload alignment slot management, etc.

[0159] The capabilities (e.g., each capability) of surgical instrument A 47005 and / or surgical instrument B 47010 may be associated with a corresponding autonomy level (e.g., capability 47015 associated with surgical instrument A 47005 may be associated with autonomy level 47030, and capability 47020 associated with surgical instrument B 470010 may be associated with autonomy level 47025). For example, they may be classified by numbers such as 1, 2, 3, etc., where 1 may represent an autonomy level with the least manual input when compared to other autonomy levels. For example, an autonomy level with the least manual input may result in the user of the surgical instrument having to perform fewer surgical tasks associated with the capability when compared to other autonomy levels. If the surgical instrument is an endoscopic cutter and the capability is to remove tissue, autonomy level 1 may result in the surgeon having to perform only control of the energy source, while autonomy level 2 may result in the surgeon having to perform control of the energy source, body orientation, and / or body position. Tasks not performed by the surgeon may be performed using autonomous functions.

[0160] Data may be generated based on the performance of surgical instruments A 47005 and / or B 47010 (e.g., via a monitoring module located at the surgical hub 47040 or via a monitoring module associated with the surgical instrument). Fig.11The data may be related to the current level of autonomy of the surgical instrument in terms of performance. For example, the data may be associated with physical measurements, physiological measurements, etc., such as Fig.10 and Fig.11 These measurements are described in more detail in U.S. Patent Application No. 17 / 156,28, filed on November 10, 2021, under the title “Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements,” the disclosure of which is incorporated herein by reference in its entirety.

[0161] For example, an indication 47035 of the data may be sent to a surgical hub 47040 where the indication may be evaluated. In an example, the indication 47035 may be sent to a cloud service (e.g., an Amazon web service) as described herein. The data may be used as input, for example, in an analysis module to check whether the performance of the surgical instrument falls within a satisfactory range. The surgical instrument may include capabilities associated with a current autonomy level (e.g., autonomy level 1). If the performance-related data falls outside of the range (e.g., across a range such as regarding Fig.11 and Fig. 9 47040 may send an indication 47035 back to the surgical instrument (e.g., surgical instrument A 47005 and / or B 47010) to switch the autonomy level associated with that capability. For example, if the performance-related data exceeds the threshold, the resected tissue may be switched from autonomy level 1 to autonomy level 2.

[0162] Fig. 9 An example of an autonomous module 47050 associated with a surgical instrument 47045 and a surgical instrument capability 47060 is shown. The autonomous module 47050 can be associated with the surgical instrument 47045. In an example, the autonomous module 47045 can be part of the software of the surgical instrument. In an example, the autonomous module 47045 can be as described with respect to Figure 8The autonomous module 47045 may track the links between the capabilities 47060 and the autonomy levels of the surgical instruments. For example, the autonomous module 47050 may reference a database (e.g., a structured query language (SQL) database), for example, via a query, that maintains the links between the capabilities 47060 and the autonomy levels at a given time (e.g., maintaining capability A running at autonomy level 1 at the current time). The database may be updated based on the autonomy level associated with the capability change. In an example, the database may link one or more tasks 47055 to the autonomy level. In this case, the capabilities 47060, the autonomy levels associated with the capabilities 47060, and the surgical tasks 47055 associated with the autonomy levels may be linked and queried by the autonomous module.

[0163] In an example, the autonomous module 47050 may include all surgical tasks associated with the surgical instrument 47045. The autonomous module 47050 may append the surgical tasks to a data structure (e.g., a list) associated with the corresponding capability 47060. The module 47050 may specify one or more surgical tasks 47055 to be performed autonomously for the surgical instrument 47045 based on the capability 47060 and the autonomy level as described herein. For example, the surgical instrument 47045 may be an endoscopic cutter. The capability of the endoscopic cutter may be to remove tissue, and the autonomy level of the capability 47060 may be 1. The autonomous module 47050 may specify that control of energy source, cutting, suturing, knob orientation, body orientation, body position, anvil jaw force, and reload alignment slot management will be performed autonomously. In this case, if the capability 47060 is switched to autonomy level 2, the autonomous module 47050 may specify that fewer tasks 47055 will be performed autonomously. For example, for autonomy level 2, the autonomous module 47055 may specify that body position, anvil jaw force, and reload alignment slot management are to be performed autonomously. Other tasks 47055 may be performed manually by the surgeon.

[0164] The surgical instrument 47045 may include capabilities 47060, where each capability 47060 may only operate at certain autonomy levels. In an example, the autonomy module 47050 may reference the rules engine to check whether the capability 47060 is allowed to operate at a given autonomy level. For example, capability A may be organ mobilization. Tissue mobilization may only allow the surgical instrument 47045 to operate at autonomy level 1 or autonomy level 3 (e.g., Fig. 9 Capability B may be anastomosis. Anastomosis may only allow surgical instrument 47045 to operate at autonomy level 2 (e.g., as shown). Fig. 9 shown).

[0165] Fig.10An example of a computer-implemented autonomous surgical system is shown. The system may include a processor associated with a surgical instrument 47065 and a processor associated with a surgical hub 47115. The surgical instrument processor may be coupled via a system bus to communications (e.g., a communications module), storage 47080, memory (e.g., non-volatile memory), management modules, actuators, sensors, input interfaces (e.g., which may obtain 47075 measurements 47070 from external sources, such as physical, physiological, and visual based measurements, such as about Fig. 9 and Fig.11 The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures including, but not limited to, a 9-bit bus, Industry Standard Architecture (ISA), Micro Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer System Interface (SCSI), or any other peripheral bus.

[0166] The surgical hub processor may be coupled to communications, storage device 47080, memory (e.g., non-volatile memory), input / output interfaces, analysis modules, and management modules via a system bus.

[0167] Each processor may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be a commercially available LM4F230H5QR ARM Cortex-M4F processor core, such as Texas Instruments, which includes 256KB of single-cycle flash or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improved execution above 40MHz, 32KB of single-cycle sequential random access memory (SRAM), a 32KB MCU loaded with Internal read-only memory (ROM) for software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which can be found in the product data sheet.

[0168] In an example, each processor may include a safety controller including two controller-based families such as TMS570 and RM4x, also known by Texas Instruments under the trade name Hercules ARMCortex R4. The safety controller may be configured to be specifically designed for IEC 61508 and ISO 26262 safety-critical applications, etc., to provide advanced integrated safety features while delivering scalable execution, connectivity, and memory options.

[0169] System memory may include volatile memory and non-volatile memory. A basic input / output system (BIOS), which contains basic routines such as transferring information between elements within a computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random access memory (RAM) that acts as external cache memory. In addition, RAM is available in a variety of forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM) enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct Rambus RAM (DRRAM).

[0170] The system may include removable / non-removable, volatile / non-volatile computer storage media, such as disk storage devices. The disk storage devices may include, but are not limited to, devices such as disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage devices may include storage media, either alone or in combination with other storage media, including but not limited to optical disk drives such as compact disk ROM devices (CD-ROM), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital versatile disk ROM drives (DVD-ROM). In order to facilitate the connection of the disk storage device to the system bus, a removable or non-removable interface may be used.

[0171] It should be understood that the system may include software that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment. Such software may include an operating system. In an example, the operating system may be associated with management modules for surgical instruments 47065 and surgical hubs 47115, respectively. An operating system that may be stored on a disk storage device may be used to control and allocate resources of a computer system. System applications may utilize the operating system to manage resources through program modules and program data stored in system memory or disk storage devices. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0172] The user can input commands or information into the system through an input device coupled to the I / O interface. These and other input devices are connected to the processor through the system bus via (one or more) interface ports. Interface ports include, for example, serial ports, parallel ports, game ports, and USB. Output devices use ports of the same type as input devices. Thus, for example, a USB port can be used to provide input to a computer system and output information from the computer system to an output device. The output device can be a surgical hub 47115. The output device can be a surgical instrument 47065. Output adapters are provided to illustrate that some output devices, such as monitors, displays, speakers, and printers, may exist in other output devices and may require special adapters. Output adapters may include, but are not limited to, video and sound cards that provide connection devices between output devices and the system bus by way of example. It should be noted that other devices or systems of devices such as remote computers can provide both input capabilities and output capabilities.

[0173] The system may be used to communicate with one or more remote computers (such as cloud computers) (e.g., as described in relation to Figure 8 The remote cloud computer can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device or other public network nodes, etc., and generally includes many or all of the elements described relative to the computer system. For simplicity, only a memory storage device with a remote computer is shown. The remote computer can be logically connected to the computer system through a network interface, and then physically connected via a communication connection. The network interface can cover communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technology may include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet / IEEE 802.3, token ring / IEEE 802.5, etc. WAN technology may include but is not limited to point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variants thereof, packet switching networks, and digital subscriber lines (DSL).

[0174] In various aspects, the computer system may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) for processing digital images. The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instructions, multiple data (MIMD) techniques to improve speed and efficiency. The digital image processing engine may perform a range of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0175] The communication connection may refer to the hardware / software used to connect the network interface to the bus. Although the communication connection is shown for exemplary clarification within the computer system, it may also be located outside the computer system. The hardware / software necessary to connect to the network interface may include internal and external technologies such as modems, including conventional telephone-grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards for exemplary purposes only.

[0176] The surgical instrument 47065 may include one or more hardware components, such as actuators, sensors, and the like. While the surgical instrument is performing a surgical procedure, data may be generated from the hardware components, transmitted to the surgical instrument's memory, and archived in the surgical instrument's storage device 47080. The data may be related to physical, physiological, visual, and the like-based conditions of the surgical instrument's performance. As described herein, the surgical instrument may receive 47075 data from an external source. The physical conditions may be related to the forces being applied by the surgical instrument (e.g., mechanical grip) and / or the kinematics of the surgical instrument 47065 (e.g., the position, velocity, and / or orientation of the instrument).

[0177] The data (e.g., an indication 47150 of the data) may be transmitted to the surgical hub 47115 via communications as described herein. The surgical hub 47115 may use the data as input into an analysis module. The analysis module may evaluate whether the data crosses a threshold (e.g., within a range as described with respect to Fig. 9 and Fig.11 The surgical hub 47115 may archive the data in a storage device 47120 (e.g., the data may be used in a subsequent surgical procedure). The data may be analyzed (e.g., by the analysis module) and an indication 47145 of the analyzed data may be sent back to the surgical instrument 47065 as input.

[0178] Fig.11An example of monitoring the execution of surgical tasks associated with a level of autonomy is shown. The monitoring module 47155 can be part of a surgical instrument, a surgical hub, or a remote device (e.g., a third-party cloud service). In an example, aspects of the monitoring module 48155 can be distributed among multiple devices (e.g., each device can be responsible for executing a certain set of instructions). The monitoring module 47155 can be as described with respect to Fig.10 The analysis module 47155 can evaluate whether the performance of the surgical instrument capabilities operating at an autonomous level is within an acceptable (eg, satisfactory) range.

[0179] Monitoring module 48155 can obtain data related to surgical instruments and / or patient 47175, such as Fig.11 The data may be generated from an external source (e.g., a wearable on a patient 47175 measuring biomarker data), and an indication of the data may be sent to the monitoring module 47155 via a message. The message may be responsive to the monitoring module 47155 requesting the data via a request message. The data may be in raw form and may be processed by a processor (e.g., as described with respect to Fig.10 The surgical hub processor or surgical instrument processor) transforms it into a form suitable for analysis.

[0180] The data may be related to the performance of the surgical procedure. For example, the data may be related to the measured physiological conditions of the patient 47175, the surgeon, and / or personnel involved in the surgical procedure. For example, the patient 47175 may be wearing a heart rate monitoring wearable device. The monitoring wearable device is described in more detail under the title "Method Of Adjusting A Surgical Parameter Based On Biomarker Measurements" in U.S. patent application US17 / 156,28 filed on November 10, 2021, the disclosure of which is incorporated herein by reference in its entirety. In this case, data related to the patient's heart rate may be obtained and used by the monitoring module. The surgeon may be wearing a wearable headband that measures pressure via a sweat sensor (e.g., by inferring the level of cortisol that the surgeon is producing). Data related to the measured physiological condition may be organized in a measurement physiological module 47170, which may be part of a monitoring module 47155.

[0181] The data may be related to the measured physical condition of the instrument, patient 47175, surgeon, and / or staff. For example, potentiometers and / or sensors may be placed on surgical instruments and data related to the movement and / or orientation of the instrument may be generated and sent to monitoring module 47155. For example, as described with respect to Fig. 9The apparatus described as autonomously performing tasks based on its autonomy level can move from position A to position B. The rate at which the apparatus moves can be calculated (e.g., based on calculating a reading from a potentiometer), and an indication of the rate can be sent to a monitoring module. In an example, a measured physical condition can be obtained via data generated by a vision-based device. The data can be organized in a measure physical module 47160. In one example, a physical and / or physiological condition can be the object of the operation. In one example, a condition other than a physical and / or physiological condition can be the object of the operation.

[0182] Monitoring module 47155 may communicate with the surgical instrument. The surgical instrument may include capabilities, and the surgical instrument may be operating at a current autonomy level (e.g., Fig.11 The automated task 47210 may be gripping, positioning, and orientation, such as Fig.11 This allows other tasks associated with firing and suturing capabilities to be performed manually (e.g., by a surgeon). Data related to automated tasks 47210 and manual tasks 47215 may be generated and sent to monitoring module 47155. The data may be organized in measurement physics module 47160, as described herein.

[0183] Monitoring module 47155 may include ideal physical module 47180 and / or ideal physiological module 47185. These modules may include ideal values ​​associated with a number of physical and physiological conditions. The ideal values ​​may be dynamic and change based on the surgical context. For example, if a surgeon is in a critical phase of a surgical procedure, the ideal value associated with the surgeon's heart rate may increase. Fig. 9 As described, ideal values ​​can be generated based on historical data. Ideal values ​​can be manually input by a surgeon or other health care professional, for example. In an example, the measured physical module 47160, the measured physiological module 47170, the ideal physical module 47180, and the ideal physiological module 47185 can be organized in different formats.

[0184] The monitoring module 47155 may compare the measured physical module 47160 and the measured physiological module 47170 to the ideal physical module 47180 and the ideal physiological module 47185, respectively. In an example, an analysis module may be included in the monitoring module 47155 and the analysis module may be responsible for evaluating the comparison. The difference between the measured module and the ideal module may be calculated and the incremental output 47200 may be generated. In an example, a weight may be assigned to each physiological condition and the weight may be considered when generating the incremental output 47200. In an example, only a subset of the measured physical and / or measured physiological conditions may be compared to the ideal condition, which may be as described with respect to Fig. 9 The determination is based on the surgical context.

[0185] The incremental output 47200 may be compared to, for example, a trigger condition (e.g., a trigger event) to assess whether the trigger condition has been met. The trigger condition may indicate whether the difference in value between the measured module and the ideal module exceeds an acceptable value and / or range. In an example, an incremental output 47200 may be generated for each physiological and / or physical condition, and each incremental output 47200 may be compared to a corresponding trigger condition.

[0186] Based on the trigger condition being satisfied, the autonomy level associated with the surgical instrument capability may be switched 47220 to another autonomy level. In the example, the autonomy level may only be switched to the autonomy level allowed for the capability, such as with respect to Fig. 9 As stated. Fig.11 As shown, the triggering condition has been met and the autonomy level at which the surgical instrument operates to perform its capabilities switches 47220 from autonomy level 1 47190 to autonomy level 2 47230. Autonomy level 2 47230 may result in fewer tasks being performed autonomously (e.g., automated tasks 47235) and more tasks being performed manually (e.g., manual tasks 47240). For example, Fig.11 As shown, when the surgical instrument is in autonomy level 2 47230, clamping can be performed autonomously and the surgeon can be caused to manually perform firing, suturing, positioning and orientation. Updates to the autonomy level can be sent to the database, such as about Fig. 9 described.

[0187] Fig.12 An example of a relationship 47245 between an error magnitude and an autonomy level is shown. In an example, an error magnitude may be associated with (e.g., linked to) a corresponding autonomy level. The error magnitude may be used as a proxy for Fig.11 For example, regarding Fig.11The monitoring module may generate an output (e.g., a delta output) based on the difference between the measured condition and the ideal condition. The output may be compared to an error magnitude (e.g., used as a threshold) to determine whether the current autonomy level results in an acceptable outcome. For example, Fig.12 As shown, autonomy level 1 may be associated with an error magnitude of 0.01%. If the output from the monitoring module (eg, which may be a numerical value) exceeds 0.01%, an indication may be sent to the surgical instrument to switch the autonomy level to, for example, autonomy level 2.

[0188] like Fig.12 As shown, the error magnitude may increase with increasing autonomy levels, where increasing autonomy levels have less autonomously performed tasks, such as with respect to Fig. 9 If the autonomy level is switched, an indication of the switch may be sent to the monitoring module, and the indication may include an updated error magnitude. For example, if the autonomy level is switched to autonomy level 2, an error magnitude of 0.1% may be sent to the monitoring module. The monitoring module may include a database, for example, about Fig. 9 and Fig.11 The monitoring module may send a message to the database to update the current autonomy level at which the surgical instrument is operating and the error magnitude associated with the updated autonomy level. The monitoring module may infer the autonomy level based on the error magnitude.

[0189] An error magnitude associated with an autonomy level may be determined. For example, a surgical hub may determine an error magnitude for an autonomy level based on historical data. The historical data may show a correlation between the likelihood of an adverse outcome and the error magnitude. The historical data may be based on an analysis of what happened in the past when an autonomous function (e.g., associated with an autonomy level) was used to perform the capability. The surgical hub may include an error magnitude module that calculates an error magnitude that minimizes the likelihood of an adverse outcome based on the historical data. In an example, the error magnitude module may determine the error magnitude based on weighing the benefits of performing a capability at an autonomous level versus performing the capability manually. For example, there may be a certain likelihood that performing the capability manually may result in an adverse outcome, which the error magnitude module may take into account when determining the error magnitude. An indication of the error magnitude may be sent to a monitoring module, such as regarding Fig.11 described.

[0190] The error magnitude associated with the corresponding autonomy level can be dynamic. For example, the surgical hub may send an acceptable error magnitude range for the autonomy level to the monitoring module. When the surgical instrument is performing this capability, surgical background data may be sent to the monitoring module, for example, from the surgical hub, the surgical instrument, or other devices associated with the surgical operation (e.g., a wearable device that the surgeon may be wearing). The monitoring module may send instructions to adjust the error magnitude based on the surgical background data. For example, the surgical background data may indicate that a key step of the surgical operation is being performed. The monitoring module may obtain this data and send instructions to reduce the error magnitude (e.g., the reduction results in crossing a threshold with fewer errors).

[0191] Fig.13 An example of the relationship between error magnitude and autonomy level is shown.

[0192] like Fig.13 As shown, a high autonomy level 47250 (e.g., which results in a large number of tasks associated with the autonomous execution of surgical instrument capabilities) can be associated with a low error magnitude (e.g., a total error magnitude). In an example, a high autonomy level 47250 can be represented as autonomy level 1, as described herein. A low autonomy level 47260 (e.g., which results in a small number of tasks associated with the autonomous execution of surgical instrument capabilities) can be associated with a high error magnitude (e.g., a total error magnitude). As described with respect to Fig.12 As described, the error magnitude may be associated with a threshold value used to determine whether to switch the level of autonomy at which the surgical instrument is operating.

[0193] As about Fig.12 and Fig. 9 The method may be based on historical data, surgical context (e.g., it may include environmental data such as the number of staff in the surgical operating room (OR), Fig.11 The error magnitude may be determined based on the surgical instrument data described above, etc. The error magnitude may be determined based on the surgical task being performed autonomously. For example, the capability of an endoscopic cutter may be to remove tissue. It may be set to autonomy level 1 to perform this capability, which may result in autonomous execution of these surgical tasks (e.g., they are tasks associated with removing tissue) such as controlling the energy source, cutting, suturing, knob orientation, body orientation, body position, anvil jaw force, and reloading the alignment slot to manage these surgical tasks. Data related to one or more of these surgical tasks (e.g., knob orientation) may be obtained by the surgical hub, and the surgical hub may use this data when determining the error magnitude. For example, the data may indicate that the knob orientation is a task that is likely to be successfully performed when completed autonomously. In this case, the surgical hub may determine a low error magnitude. In an example, the error magnitude may be sent to a monitoring module, such as regarding Fig.12As described, the monitoring module can infer the autonomy level that the instrument is using. This data may indicate that suturing is a task that is unlikely to be successfully performed when completed autonomously. In this case, the surgical hub can determine a high error magnitude. Data related to multiple tasks can be used, for example, by the surgical hub to determine error magnitudes. For example, data from knob orientation and data from suturing can both be sent to the surgical hub. The surgical hub can weigh the two sets of data and determine a medium autonomy level 47255 to be used, for example, because knob orientation is a task that is likely to be successfully performed when completed autonomously, and suturing is a task that is unlikely to be successfully performed when completed autonomously. Error magnitudes can be determined for each task associated with the capability to be performed. For example, the surgical hub can determine a knob orientation error magnitude and a suturing error magnitude, and the knob orientation error magnitude can be as described with respect to Fig.12 The stitching error magnitude may be used when evaluating whether the autonomy level associated with the knob orientation will switch.

[0194] Fig.14 An example of the relationship between autonomy level, machine learning, and surgical tasks is shown.Autonomy level and automation level are used interchangeably herein.

[0195] As described herein, autonomously performing surgical tasks associated with surgical instrument capabilities may involve the use of a machine learning framework. The machine learning framework is described in more detail under the title "Method for Surgical Simulation" in U.S. patent application US17 / 332,593 filed on May 27, 2021, the disclosure of which is incorporated herein by reference in its entirety. For example, inputs (e.g., parameters) associated with a surgical task may be sent to a machine learning model 47275, and the machine learning model may be trained 47280 to autonomously perform a surgical task 47285. The machine learning model 47275 may involve determining an autonomous level 47265 (e.g., autonomous level 1) at which the execution of the surgical task 47285 should be performed. Data related to the autonomous level 47265 may be generated and sent to the machine learning model 47275 in the form of feedback 47270, wherein the model 47275 may adjust the inputs associated with the surgical task 47285 and / or change the autonomous level 47265 of the current operation of the surgical instrument 47285. The machine learning model 47275 may be located locally on the surgical instrument (e.g., a module) or on a remote device.

[0196] Dynamic variables associated with autonomously performing surgical tasks 47285 can be determined by machine learning algorithms based on, for example, Fig.11 For example, the autonomy level 47265 associated with surgical task 47285 may be updated based on the performance metrics described. Fig.12The machine learning model 47275 may use a real-world dataset of previous surgeries when determining whether an error magnitude has been crossed and / or when determining an autonomy level 47265 associated with a surgical task 47285. The real-world dataset of previous surgeries may be an aggregation of surgeries performed by a surgeon, surgeries from the facility, and / or a compilation of surgeons using a hub within the same network.

[0197] The machine learning algorithms may be updated by data, for example, from the cloud and / or remote systems that may compile best practices, regional data regarding surgical procedures, and / or worldwide results and procedures used from any number of other facilities around the world.

[0198] Real-world information may be derived from procedure results (eg, from a region, population, etc.), and / or may be an interpolation and / or aggregation of sub-biomarker metrics and results.

[0199] For example, a GAN-based machine learning model 47275 (GAN model) can be trained using past surgical data. The GAN model can model the data pattern of a given surgical step and can autonomously perform a list surgical task 47285. The GAN model can model the probability distribution of surgical tasks 47285 present in past surgical data. That is, when the GAN model generates a surgical task 47285 from the list of possible surgical tasks 47285, the surgical task 47285 is generated with a probability according to the probability distribution.

[0200] Machine learning may be part of a technology platform known as cognitive computing (CC), which may comprise various disciplines such as computer science and cognitive science. CC systems may be able to learn at scale, reason purposefully, and interact naturally with humans. CC systems may be able to perform surgical tasks autonomously.

[0201] The output of the training 47280 process of machine learning can be a model 47275 for predicting the results of a new data set. For example, a linear regression learning algorithm can be a cost function that minimizes the prediction error of the linear prediction function during the training process by adjusting the coefficients and constants of the linear prediction function. When the minimum value can be reached, the linear prediction function with the adjusted coefficients can be considered to be trained and constitutes a model 47275 that has produced the training 47280 process. For example, a neural network (NN) algorithm for classification (e.g., a multilayer perceptron (MLP)) may include a hypothesis function represented by a network of node layers, which is assigned a bias and interconnected with weight connections. The hypothesis function can be a nonlinear function (e.g., a highly nonlinear function), which may include linear functions and logic functions nested together, where the outermost layer consists of one or more logic functions. The NN algorithm may include a cost function to minimize the classification error by adjusting the bias and weights through a process of feedforward propagation and backward propagation. When a global minimum can be reached, the optimized hypothesis function of the layer with its adjusted biases and weights can be considered to be trained and constitute the model 47275 that has produced the training process 47280.

[0202] Fig.15 An example of determining the level of autonomy of a surgical instrument in performing a surgical task is shown. Fig. 9 As described above, surgical tasks can be associated with capabilities of surgical instruments (eg, tissue resection). Surgical steps can be associated with capabilities.

[0203] The surgical procedure may follow a surgical plan 47290, which may outline the surgical steps to be performed (e.g., surgical step 1 47295, surgical step 2 47300, surgical step 3 47305, surgical step 4 47310, and surgical step K 47315). These surgical steps may be performed sequentially. In an example, these surgical steps may be performed in parallel. A surgical hub or other device may obtain the name of the surgical procedure to be performed and generate a surgical plan based on historical data related to the success of previous surgical procedures or previous simulations of surgical procedures (e.g., performed at a facility). The surgical hub may take data (e.g., additional data) into account when generating a surgical plan, such as the surgeon's experience level.

[0204] One or more surgical steps (e.g., each surgical step) may include a surgical task to be performed by a surgical instrument. Fig.11 As described, one or more of these tasks may be performed autonomously and one or more tasks may be performed manually, which may be determined based on the autonomy level associated with the capability (e.g., surgical step). In an example, for example, if the capability is associated with the highest autonomy level (e.g., Fig.12If the surgical team is associated with the “full autonomy” level shown in Figure 1, all surgical tasks can be performed autonomously.

[0205] Machine learning models may be used to determine the level of autonomy of a surgical step (e.g., machine learning model 47320 associated with surgical step 1 47295 and machine learning model 47340 associated with surgical step 4), such as with respect to Fig.14 As described. In an example, the parameters of the machine learning model may be input by a user 47355. The machine learning model 47320 may output that for surgical step 1 47295, tasks 1 47360 to N 47375 will be performed autonomously and tasks N+1 47365 to Z 47380 will be performed manually (e.g., by a surgeon or another surgical staff member). The machine learning framework may receive feedback on the execution of the surgical step. If the execution crosses a threshold, the machine learning model may obtain a message that the autonomy level will switch. In this case, the model may determine an updated autonomy level based on feedback on the execution of the surgical step, which may include one or more of real-time surgical data, user data, surgical environment data, surgical instrument data, task data, or historical data. The update may occur during the surgical step. The update may occur based on a transition from one surgical step to another surgical step (e.g., during a transition from surgical step 1 to surgical step 2). The updated autonomy level may be associated with different groups of surgical tasks to be automated (e.g., autonomously performed). For example, based on the transition to surgical step 4 47310, tasks 1 47360 through X 47400 may be autonomously performed, and tasks X+1 47395 through L 47405 may be manually performed. A message 47410 (e.g., Simple Notification Service (SNS)) regarding the updated autonomy level may be sent to the surgeon or other surgical staff member. Surgical tasks to be performed autonomously and manually may be sent via the message.

[0206] Fig.16 An example flow chart 47415 for automating a surgical task associated with an autonomy level is shown. At 47420, the device may receive an indication of a surgical task to be performed using a surgical instrument. The capabilities of the surgical instrument may be associated with an automation level.

[0207] At 47425, the apparatus may monitor performance of a surgical task with the surgical instrument operating at a first automation level associated with the automation level. The capabilities of the surgical instrument may include a set of surgical instrument tasks, and the automation level may be associated with automating one or more surgical instrument tasks in the set of surgical instrument tasks.

[0208] At 47430, the device may detect a triggering event associated with the performance of a surgical task and, for example, switch the operation of the surgical instrument from a first automation level to a second automation level associated with the automation level based on the triggering event. The performance of the surgical task may be based on real-time surgical data. The real-time surgical data may include one or more of the following: user data, surgical environment data, surgical instrument data, task data, historical data, etc.

[0209] At 47435, the apparatus may detect a triggering event by comparing the performance of the surgical task to a triggering event threshold, and may switch the operation of the surgical instrument from a first automation level to a second automation level based on the triggering event. The second automation level may be associated with automating fewer surgical instrument tasks when compared to the first automation level. Monitoring the performance of the surgical task may include comparing one or more of the real-time surgical data to corresponding ideal surgical data.

[0210] In an example, the device may operate at a first automation level in the automation levels associated with the surgical task. The device may obtain an indication to switch to a second automation level in the automation levels associated with the surgical task. The indication may be based on detecting a trigger condition. The device may operate at the second automation level in the automation levels associated with the surgical task based on the indication. The indication may be based on monitoring execution of the first automation level. Execution of the first automation level may be based on real-time surgical data.

[0211] Autonomous decision making and assistance may be provided. Automatically determining the level of autonomy (eg, level of automation) from a predefined set of options may be based on identifying the circumstances of the surgical task (eg, using situational awareness).

[0212] The intelligent medical device determination of the level of automation may be based on the monitored circumstances of the surgery. For example, a powered adaptive medical device control algorithm for controlling the functions of a medical device may include automation of variable quantities. Monitoring of the instrument, surgeon, and / or patient may control the amount or level of automation (e.g., control algorithm) without direct user control. In an example, the level of automation may be based on one or more of the following: the capabilities of the device, the connection of the device to other devices, the presence of multiple personnel, or detection of aspects of its state and / or configuration.

[0213] A classification of automated tasks (e.g., a set of available selectable options) may be requested by a surgical instrument (e.g., a request message may be sent to a surgical hub), and an assessment of complexity may be used to automatically determine an appropriate level of automation for the surgical instrument, such as to perform a surgical task.

[0214] The features described herein may lead to more autonomy. For example, the product may be released from automatic operation based on maintenance. The level of autonomy may be determined based on the skill level of the user (e.g., surgeon). For example, a user with less experience (e.g., resident surgeon) may trigger the use of a lower level of automation, such as to ensure the correct operation of a surgical instrument. For example, a user with less experience (e.g., resident surgeon) may have to pull an endoscopic cutter in an emergency for a surgical task. The surgical hub and / or surgical instrument may recognize that the user's skill level is inexperienced (e.g., novel) and may set automation to minimize options and maximize ease of use in the case of limited user control activation. The level of autonomy may be associated with user selectable options, and a user with less experience may have fewer selectable options than an experienced user (e.g., selectable options are locked and automatically grayed out). A more advanced user may be given more automated options than a user with less experience (e.g., surgical tasks caused by using a higher level of automation).

[0215] The level of autonomy may be based on the user's field of view. For example, if the surgical procedure is moved outside the field of view (e.g., the laparoscopic field of view), autonomous instrument control may be adjusted. For example, during surgery, the surgeon may have multiple ports connected to multiple different instruments to perform the intended tasks, and may have a laparoscopic camera with a limited / restricted field of view. Throughout the surgical procedure, the surgeon may switch between instruments and / or move instruments to increase proximity. In this case, one or more of these instruments may no longer be in the field of view. The instrument may activate an autonomous mode in which the device functions may not be activated until it is moved back into the field of view.

[0216] The device (e.g., surgical instrument) orientation may be autonomously controlled. For example, if the instrument is out of the field of view and the surgeon attempts to bring the instrument back into the field of view, the device end effector may autonomously control itself so that it can rotate and / or orient itself out of contact with other structures / tissues / organs until it is back in the field of view, which may prevent unintended motion.

[0217] Autonomous instrument control may be based on the surgeon's focus or viewing angle, which may be fixed on a portion of the screen. For example, during a surgical procedure, the surgeon may be viewing from a laparoscopic camera. The surgeon may be disoriented and / or fixed on a functional task (e.g., his / her focus may be fixed on a portion of the screen while maneuvering or creating access to a target site). In this case, the surgical instrument may switch to an autonomous mode that will limit certain functions until the surgeon's eyes are redirected to the instrument. In this case, the instrument may determine an autonomous mode that will result in no contact with other structures. For example, the autonomous mode may allow the end effector to move or orient autonomously as the surgeon translates the device into the focus area of ​​the screen.

[0218] The facility operator and / or surgeon may choose to limit autonomy customization based on risk level. The autonomy level of the assisted surgical procedure may be set / controlled (e.g., in advance). For example, this may be based on administrative approval, safety considerations, and / or patient risk. The facility operator may choose to use default automation, which may enable less experienced surgeons to achieve more repeatable results. The facility operator may choose to disable or deactivate the available autonomy level operations, for example, until it is verified that the behavior of these levels is consistent with the operator's approach to surgical intervention and / or outcome. This may prevent a device from being brought into a facility and resulting in a poor outcome due to incompatible autonomous functions being used.

[0219] Adjusting the level of autonomy may be based on the task and / or patient risk. The laparoscopic ultrasound device may have multifunctional uses, such as one or more of the following: coagulation, cutting, dissection, or grasping. Based on the intended function, target area, and / or surgical task, the level of autonomy may be controlled according to the intended function and / or patient risk, for example, to control the allowed operations through a combination of autonomous systems and / or human activation. In the example, the higher the risk to the patient, the execution of the task may be transferred from an autonomous state to a hybrid state, and the surgeon must approve the operation before performing the operation because the operation may bring greater negative results to the patient. In this case, if an unexpected result occurs, the surgeon may be prepared to react (for example, a computer-controlled system may have more precise control, but a human / surgeon may anticipate the occurrence of problems and adjust the response to complete the task). For example, the patient risk may be level 1. The surgical instrument may be used for grasping and tissue manipulation. Based on level 1 risk, the surgical instrument may allow full autonomous opening and closing of the jaws. For example, the patient risk may be level 2. The surgical instrument may be used for cutting and dissection. Based on level 2 risk, the surgical instrument may allow full autonomy of the jaws and energy activation when on non-vascular tissue but may not be on fat tissue. In an example, the patient risk level may be 3. The surgical instrument may be used for vessel sealing and / or coagulation. Based on a 3 risk level, the surgical instrument may allow full autonomy of the jaws, but energy activation may be controlled / applied by the surgeon. This may result in less automation and more discrete operations. The autonomy level may be based on surgical complexity and / or patient-specific influences (e.g., comorbidities). The autonomy level may be based on detected issues.

[0220] A facility may limit device automation (eg, automation level) of a process, for example, until the facility has validation of use (eg, device functionality is as expected and is cost-effective).

[0221] The level of autonomy may be based on safety risks, improper operation of the surgical instrument, and / or detected surgical instrument problems (e.g., switched based on the foregoing). For example, safe energy activation may be used. For ultrasonic and / or RF energy devices, keeping the jaws and active electrodes clean and free of debris throughout the procedure may prevent tissue accumulation, which may cause unexpected generator errors and / or reduced performance. To complete the task, the surgeon may remove the device from the patient, and the physician and / or instrument nurse may use a sponge to clean the jaws. As a safety precaution, when the jaws are sensed to be cleaned, the device may autonomously activate a safe mode to ensure that energy activation is not activated while cleaning. In an example, for a harmonic device, if the blade is accidentally activated while cleaning the blade when using hemostasis and / or the blade contacts something while being cleaned and / or moved back to the patient, this may cause scratches, nicks, and / or notches in the blade, which may increase the likelihood of premature blade failure. Having the power supply autonomously deactivated when the device is not in the patient may minimize these problems. For example, a safe jaw closure may be used. For surgical procedures, a device with an end effector may be closed when passing through a trocar. The device can autonomously close the jaws before being removed or inserted into the trocar. Thermal damage can be used. Energy devices like ultrasonic scalpels (e.g., ACE series) can reach temperatures exceeding 200°C when performing the intended task or after deactivating the energy button. These devices may take longer to cool. The system can autonomously control the allowed movement of the jaws and / or limit movement until the acceptable temperature of the jaws is met, thereby ensuring that adjacent tissues are not inadvertently affected by thermal damage.

[0222] Depending on the settings and / or options available, having selectable levels of automation can result in more or less autonomy. Facility operators and / or users can use more advanced features (e.g., associated with higher levels of autonomy) in specific settings. For example, the level of autonomy can be determined by the use of intelligent software modules, more comprehensive control programs, and / or the capabilities of the hardware. Users of higher-level functionality can select a more constrained level of autonomy based on their needs (e.g., a teaching facility can use a higher level of autonomy to ensure the safety of less experienced surgeons, or a regional facility can disable an autonomous function because it is deemed not to be beneficial to patients in that area or it may cause more challenges than it can solve).

[0223] A tiered approach may be used to assess the risks associated with a surgical task, which may include one or more of the following: accepting the risk, avoiding the risk, transferring the risk, or reducing the risk.

[0224] The device may be identified within a larger digital ecosystem. The device may be detected by other systems that are capable of integrating with or communicating with the device and / or system.

[0225] The level of autonomy may be based on the presence of a collaborating device or system. For example, a first smart device may detect the presence of a second smart device or surgical hub that may initiate an automatic communication interaction. Once the smart devices are communicating, the type and / or configuration of the second smart hub may enable automatic updating or transmission of operating parameters of the first device, which may enable updating of the system. Once the first device is operated, it may automatically update the second smart hub with information about its use, which in turn may be automatically compiled with other information from other devices, and automatically parsed and propagated to other systems that use some portion of the data. This distributed data may be used when determining the autonomy level of a surgical instrument within a collaborative system. Product detection may be used as a trigger condition for executing the features described herein. For example, one or more of the following may be trigger conditions: the presence of a smart device near a smart hub; the presence of a smart component within a smart device (e.g., an RFID within a smart stitcher); the presence or proximity of a collaborative compatible device (e.g., a smart proximity sensing range or range accessory, and the presence of a smart device with an integrated fiducial marker); the first hub being within range of a second hub, a smart OR, a network gateway, and / or a communications backplane generator; or the first imaging system being within range of a second imaging system.

[0226] The ability of a system to measure and / or detect information to operate at an autonomous level may be determined.

[0227] HCPs in the room who are able to interact with or operate the device may be determined. For example, a sufficient number of appropriate users may be determined, which may include one or more of the following: area of ​​expertise and / or job function (e.g., surgeon, anesthesiologist, scrub technician, circulating nurse, etc.); experience level (e.g., total time in the medical field, time / workload in a given specialty, time / workload performing a given procedure, time / workload operating a specific piece of equipment, and / or certification and training); skill level (e.g., detected by prior operations, device use, results, etc.); or manually entered into the system by a user / administrator. Who is in the room may be determined using one or more of the following: manual entry; manual tag scanner; room badge scanner (e.g., automatically checks and updates who is in the room); facial recognition, etc.

[0228] The level of autonomy may be based on one or more of the following: There are not enough people in the room, which defaults to automated operation by a limited number of users. For example, if there are not enough staff, the level of automation may be determined (e.g., adjustments to complete the surgery). In an example, the surgery (e.g., the tasks of the surgery) may be updated, which may result in greater efficiency as adjustments increase the level of automation.

[0229] Notification may be provided of reasoning as to why a certain level of automation has been selected, which may be used to change the selection criteria to be used when selecting an automation level. The user may be notified of variables that affect the level of automation. For example, if there are not enough people in the room, the surgical instrument may default to manual operation (e.g., or default to autonomous operation).

[0230] The determination of the trigger conditions and / or the autonomy value may be based on monitored and / or calculated data feeds. During the execution of an autonomously executed task, there may not be a good autonomy level option. In this case, it may be determined how to adjust its autonomy level. For example, it may be determined to safely change from one autonomy level to a lower autonomy level. In an example, it may be determined to complete the current task and stop (e.g., not proceed to the next task). In an example, it may be determined to terminate immediately.

[0231] The magnitude of the error in the detected information can be used as a basis for determining the level of autonomy. For example, when the error level increases, the level of autonomy decreases. The error magnitude may include a criticality based on the cumulative total error of the system and / or a single error. In an example, a single error with a high criticality may reduce the autonomy of the system more than a plurality of small or insignificant errors. There may be some errors that partially eliminate the autonomy of the system. For example, a highly autonomous equipment system such as a surgical robot may have multiple arms connected to it. During the start-up test sequence, a feedback sensor of one of the arms may be used to detect that an error has occurred in the arm based on output data and feedback. It may be determined that the features of the arm are not operating correctly. The system may eliminate the autonomous activity of the arm while maintaining the autonomous activity of other arms. There may be some errors that completely eliminate the autonomy of the system. For example, a highly autonomous equipment system such as a surgical robot may use a host to process the position. Although there is redundancy in some aspects (e.g., power supply), there may be no redundancy in the central processor. If the processor fails, all autonomy of the system may be removed. There may be some errors that have no effect on the autonomy level of the system. For example, a highly autonomous system such as a surgical robot may rely on GPS signals to calculate the time and date and country for record keeping. The GPS signal may be lost and errors may occur. There may be a 24 hour retention period before which data is still considered accurate. In this case, although errors may occur, there may be no impact on functionality or autonomy levels.

[0232] In an example, the true cumulative error may not be calculated. In this case, specific functions and features related to autonomy may be removed according to the associated failure mode. Each potential autonomous function of the device may be mapped to one or more corresponding physical / software functions of the device.

[0233] A user controlled input actuator may be provided. The autonomy kill switch may include one or more of: a button that removes all autonomy from the system, e.g., regardless of the current step or state of the system; a button that removes autonomy from the system, e.g., removes autonomy after the system has completed its current activity; a multi-state button (e.g., state one of the button is for allowing autonomy to complete its current activity, and state two of the button is for immediately removing all autonomy regardless of the current activity); or a configurable button (e.g., the button may be configurable for how it should operate, such as to remove autonomy immediately or to remove autonomy upon completion of an activity).

[0234] A sensed event associated with the device may be used, which may include one or more of the following energy events: initial sensed force contact, rate of force increase, impedance threshold, thermal diffusion / damage, or jaw temperature. For example, when the jaws are closed, the first timing of the current through the motor may be sensed and may be used to determine tissue height. If it is a thickness determined for the end effector, the motor speed may be changed based on the threshold. Trigger conditions may be used to cause a change in autonomy. For example, an overly thick condition may cause the device to stop automation and request user input.

[0235] Impedance can be used as a trigger condition. For example, when energy is activated in the jaws, the generator can monitor the impedance of the tissue and / or vessel to determine when the energy should be turned off. The product code can be indicated for the maximum container sealing size (e.g., typical ≤5mm or ≤7mm size). The energy algorithm can take into account impedance over time. If the energy activation cycle is too short compared to the normal cycle, this can indicate and / or signal damaged and / or diseased tissue and indicate / notify the user to take over activation control. If the energy activation cycle is longer than the normal cycle, this can indicate and / or signal that the tissue is fat and / or a larger vessel size than the indicated product, and the responsibility can be transferred to the surgeon. Tissue accumulation on the jaws can change the impedance control. When this condition occurs based on the calculated time versus the actual time, the user can be notified to clean the jaws. In an example, Harmonic 7 can only operate at power level 3 for vessels with a diameter of ≤5mm, and operate in advance hemostasis mode for vessel sizes of ≤7mm. In this case, the system can autonomously select / change the power level based on the identified vessel size. When the vessel exceeds these sizes or is between sizes, the system can request from the user which power level to continue operating at.

[0236] Thermal diffusion can be used as a trigger condition. During energy activation and vessel sealing, the criteria that can be monitored together with hemostasis are thermal damage and / or thermal diffusion. When using energy-based technologies, temperature and / or lateral thermal damage may be a consideration for surgeons. Surgeons may worry about direct contact or visually unrecognizable energy transmission causing damage to nearby structures. Surgeons may worry about the potential impact of tissue damage on inflammatory response and overall patient recovery. Monitoring thermal diffusion during energy activation as well as tissue type, thickness, power level and / or clamp pressure can lead to calculation of actual value versus nominal value, which can be used to notify the user and / or autonomously adjust power settings and / or clamp pressure (e.g., Harmonic ACE+72.54+ / -0.48mm, which refers to the mean and standard deviation). When autonomously adjusted, preclinical comparisons of goat vessel sealing can be used. Thermal damage can be caused by heat generated by advanced energy devices and may be a key component of vessel sealing and / or tissue transection. The temperature reached by the instrument may depend on multiple variables, including tissue type, tissue thickness, energy used and / or power settings. Advanced energy devices may reach instrument temperatures of at least 100°C during activation on tissue. Due to tissue conditions, there may be situations where the temperature of the ultrasonic device may reach a range exceeding 100°C. When the jaws reach this temperature, the surgeon may inadvertently come into contact with tissue and cause trauma. When this occurs, a wait time may be triggered until the jaw temperature decreases to an acceptable temperature. In this case, if contact is made with tissue, no trauma will occur, or if the physician attempts to move the device, autonomous control may take over limited control (e.g., if the physician moves the device distally with a faster and / or larger motion than allowed, autonomy may slow down and / or reduce that displacement to ensure that the high temperature jaws do not come into contact with tissue and / or unintended treatment areas).

[0237] The suturing device can be used as a trigger condition. For example, unexpected joint forces can be used.

[0238] Automated risk determination levels may be provided. User selections may be compared to a baseline or global user selection level of other facility users to determine the level of authority associated with the autonomy level. Active surgeon selections may be monitored to control autonomous selections. Set uncertainty levels or pre-selected user risk levels may be used to determine acceptable risk levels in their decisions. Previous actions or outcomes may be used to determine what the effective risk level may be.

[0239] A risk matrix may be provided that the system uses to determine appropriate risk. These matrices may include medical or patient (e.g., claims history, such as the International Classification of Disease, 10th Revision, Clinical Modification (ICD010-CM), Hierarchical Disease Categories (HCC), Electronic Health Records ('HER-Health Information Technology (HIT) databases, etc.).

[0240] The overall risk determination may be a combination of risks associated with the patient, the device, and the procedure.

[0241] Adjustments to the level of autonomous functionality may be based on historical user controls, prior autonomous actions, and / or historical results resulting from prior autonomous determinations. Autonomous failures and / or user overrides / assistances may reduce the level of autonomy and / or the frequency of suggested autonomous actions. Undesirable results may adjust the degree of autonomous engagement (e.g., if user error caused the situation, the degree of autonomous engagement may be reduced compared to if autonomous action resulted). Historical user interactions that resulted in confusion, misuse, or usage instruction issues may result in the system assisting with autonomous prompts and / or controls to minimize future usage issues, such as delays in procedure counts.

[0242] Aggregation of data may be used to adjust the autonomy level. For example, a weighted comparison may be used to decide how to react to historical information. Criticality of the fault, local frequency, uniqueness of the fault, etc. may be used. Uniqueness may include competing motor faults and / or a complete reset of the system (e.g., RF interference system operation resulting in a complete restart to continue use).

[0243] The following is a list of numbered embodiments that may or may not be claimed:

[0244] 1. An apparatus for changing the automation level of a surgical instrument used to perform a surgical procedure, the apparatus comprising:

[0245] A processor configured to:

[0246] receiving an indication of a surgical procedure to be performed using a surgical instrument, the surgical procedure comprising a plurality of surgical tasks to be performed using the surgical instrument;

[0247] monitoring performance of surgical tasks in the plurality of surgical tasks with the surgical instrument operating at a first level of automation;

[0248] detecting a triggering event associated with the performance of the surgical task; and

[0249] The operation of the surgical instrument is switched from the first automation level to a second automation level based on the triggering event.

[0250] Advantageously, a device for varying the level of automation of a surgical instrument used to perform a surgical procedure enables the level of automation to be increased or decreased based on monitored performance of a surgical task of the procedure to reduce the risk of an adverse surgical outcome for the task or procedure and thereby ultimately improve the surgical outcome of the surgical procedure performed.

[0251] For example, the outcome of tissue resection may correspond to the integrity of the seal line, wherein less seal leakage equates to a more positive outcome than more seal leakage. The outcome of the surgery may also correspond to the number of difficulties that occurred during the surgical procedure, which may include, for example, instrument failure, seal line leakage, staple line misfiring, etc.

[0252] 2. An apparatus according to embodiment 1, wherein the automation level corresponds to the number of automated surgical tasks among the multiple surgical tasks.

[0253] By varying the amount of surgical tasks that are automated, the risk to the patient and the risk of an adverse surgical outcome can be reduced.

[0254] 3. An apparatus according to embodiments 1 or 2, wherein the second level of automation is associated with automating fewer surgical tasks when compared to the first level of automation.

[0255] Thus, switching the automation level from the first level to the second level causes one or more surgical tasks to be manually performed after the switch, which surgical tasks had been performed autonomously before the switch.

[0256] 4. An apparatus according to embodiments 2 or 3, wherein automation of surgical tasks is associated with zero manual input from the surgeon.

[0257] 5. An apparatus according to embodiments 2 or 3, wherein automating surgical tasks is associated with reduced manual input from the surgeon.

[0258] 6. An apparatus according to any of the preceding embodiments, wherein the execution of the surgical task is based on real-time surgical data.

[0259] By using real-time surgical data to monitor the performance of surgical tasks, trigger events can be detected and responded to in real time, making it possible to change the level of automation during the current surgical procedure, such as by changing whether the monitored surgical task is automated and / or changing whether one or more of the subsequent surgical tasks of the surgical procedure are automated, which can increase the chances of a positive surgical outcome for the surgical procedure.

[0260] 7. An apparatus according to embodiment 6, wherein the real-time surgical data includes at least one of the following: user data, surgical environment data, surgical instrument data, and task data.

[0261] For example, surgical instrument data may include physical conditions measured from a surgical device performing a surgical task.User data may include measured physiological conditions of a patient, surgeon, and / or personnel involved in a surgical procedure.

[0262] 8. An apparatus according to any of the preceding embodiments, wherein the processor is further configured to detect the trigger event by comparing the performance of the surgical task to a trigger event threshold.

[0263] 9. An apparatus according to one of embodiments 6, 7 or 8, wherein monitoring the execution of the surgical task includes: comparing one or more of the real-time surgical data with corresponding ideal surgical data, and detecting the occurrence of a trigger event when the comparison exceeds a trigger event threshold.

[0264] 10. An apparatus according to embodiment 9, wherein the corresponding ideal surgical data is generated from historical data of the surgical task.

[0265] Ideal surgical data may be generated from historical surgical data associated with positive or ideal outcomes for surgical tasks. If current surgical data deviates from historical surgical data associated with positive outcomes for surgical tasks, this indicates that the surgical outcome of the current surgical procedure may not be as successful / positive as the outcome associated with the historical surgical tasks. By responding to a triggering event by changing the level of automation (which change may include changing whether the current task and / or subsequent tasks are automated), the chances of obtaining a successful surgical outcome may be improved.

[0266] 11. A device according to any of the preceding embodiments, wherein the surgical procedure includes resecting tissue and the multiple surgical tasks include one or more of the following: controlling energy source, cutting, suturing, knob orientation, body orientation, body position, anvil jaw force, and reloading alignment slot management.

[0267] 12. A device according to any of the preceding embodiments, wherein the surgical device includes a memory that stores the multiple surgical tasks associated with the surgical procedure and the surgical tasks to be automated for each autonomy level.

[0268] 13. A device according to embodiment 12, wherein the surgical instrument is suitable for performing two or more surgical procedures, the surgical instrument performs a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure, and wherein the surgical device includes a memory that stores the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and wherein the memory also stores associations between the surgical tasks to be automated for each autonomy level of each surgical task in the first plurality of surgical tasks and the second plurality of surgical tasks.

[0269] The surgical instrument may be used in a plurality of procedures, each procedure comprising a different plurality of surgical tasks, and the apparatus may be configured to be able to vary the level of automation of the instrument used to perform any of these procedures.

[0270] 14. The device of any preceding embodiment, wherein the device is a surgical hub.

[0271] 15. A computer-implemented method for providing commands or suggestions to change the automation level of a surgical instrument used to perform a surgical procedure, the method comprising:

[0272] receiving an indication of a surgical procedure to be performed using a surgical instrument, wherein the surgical procedure includes a plurality of surgical tasks to be performed using the surgical instrument;

[0273] monitoring performance of a surgical task in the plurality of surgical tasks with the surgical instrument operating at a first level of automation in the plurality of levels of automation;

[0274] detecting a triggering event associated with the performance of the surgical task; and

[0275] A command is provided to the surgical instrument or a suggestion is provided to the user to switch operation of the surgical instrument from the first automation level to a second automation level based on the triggering event.

[0276] Advantageously, the computer-implemented method may monitor the performance of surgical tasks performed by an instrument during a surgical procedure and, based on a triggering event, provide commands to the instrument or provide recommendations to a user to change the level of automation of a surgical instrument used to perform the surgical procedure such that the level of automation can be increased or decreased to reduce the risk of an adverse surgical outcome of the procedure and thereby ultimately improve the surgical outcome of the surgical procedure performed.

[0277] For example, the outcome of tissue resection may correspond to the integrity of the seal line, wherein less seal leakage equates to a more positive outcome than more seal leakage. The outcome of the surgery may also correspond to the number of difficulties that occurred during the surgical procedure, which may include, for example, instrument failure, seal line leakage, staple line misfiring, etc.

[0278] 16. A method according to embodiment 15, wherein the automation level corresponds to the number of automated surgical tasks among the multiple surgical tasks.

[0279] 17. A method according to embodiment 16, wherein the second automation level is associated with automating fewer surgical tasks when compared to the first automation level.

[0280] 18. A method according to any one of embodiments 15 to 17, wherein monitoring the performance of the surgical task is based on real-time surgical data.

[0281] By using real-time surgical data to monitor the performance of surgical tasks, trigger events can be detected and responded to in real time, making it possible to change the level of automation during the current surgical procedure, such as by changing whether the monitored surgical task is automated and / or changing whether one or more of the subsequent surgical tasks of the surgical procedure are automated, which can increase the chances of a positive surgical outcome for the surgical procedure.

[0282] 19. A method according to embodiment 18, wherein the real-time surgical data includes at least one of the following: user data, surgical environment data, surgical instrument data, and task data.

[0283] For example, surgical instrument data may include physical conditions measured from a surgical device performing a surgical task.User data may include measured physiological conditions of a patient, surgeon, and / or personnel involved in a surgical procedure.

[0284] 20. A method according to embodiments 18 or 19, wherein monitoring the execution of the surgical task includes: comparing one or more of the real-time surgical data with corresponding ideal surgical data, and detecting the occurrence of a trigger event when the comparison exceeds a threshold.

[0285] 21. A method according to embodiment 20, wherein the corresponding ideal surgical data is generated from historical data of the surgical task.

[0286] Ideal surgical data may be generated from historical surgical data associated with positive or ideal outcomes for surgical tasks. If current surgical data deviates from historical surgical data associated with positive outcomes for surgical tasks, this indicates that the surgical outcome of the current surgical procedure may not be as successful / positive as the outcome associated with the historical surgical tasks. By responding to a triggering event by changing the level of automation (which change may include changing whether the current task and / or subsequent tasks are automated), the chances of obtaining a successful surgical outcome may be improved.

[0287] 22. A method according to embodiment 21, wherein the method further comprises generating the corresponding ideal surgical data.

[0288] 23. A method according to any one of embodiments 15 to 22, wherein the method includes retrieving from a memory the multiple surgical tasks associated with the surgical procedure and the surgical tasks to be automated for each autonomy level.

[0289] 24. A method according to embodiment 23, wherein the surgical instrument is suitable for performing two or more surgical procedures, the surgical instrument performs a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure, and wherein the method includes retrieving from the memory indications of the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and retrieving from the memory associations between the surgical tasks to be automated for each autonomy level of each surgical task in the first plurality of surgical tasks and the second plurality of surgical tasks.

[0290] The surgical instrument may be used in a plurality of procedures, each procedure comprising a different plurality of surgical tasks, and the apparatus may be configured to be able to vary the level of automation of the instrument used to perform any of these procedures.

[0291] 25. A computer program comprising instructions which, when a computer executes the program, cause the computer to perform the method according to any one of embodiments 15 to 24.

[0292] 26. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of embodiments 15 to 24.

[0293] 27. An apparatus for performing multiple surgical tasks during a surgical procedure, the apparatus comprising:

[0294] A processor configured to:

[0295] Operate at the first automation level;

[0296] Obtaining an indication to switch to a second automation level; wherein the indication is based on detecting a trigger condition; and

[0297] Operating at a second automation level based on the indication.

[0298] Advantageously, a device capable of operating at different levels of automation allows for changes in automation, enabling a procedure to be more or less automated to reduce the risk of an adverse surgical outcome for the procedure and thereby ultimately improve the surgical outcome of the surgical procedure performed.

[0299] 28. The apparatus of embodiment 27, wherein the first automation level and the second automation level differ in the number of automated surgical tasks of the surgical procedure.

[0300] 29. An apparatus according to embodiment 27 or 28, wherein the indication is based on monitoring the performance of the surgical task among the plurality of surgical tasks at the first automation level.

[0301] 30. An apparatus according to embodiment 29, wherein the execution of the surgical task at the first automation level is based on real-time surgical data.

[0302] By using real-time surgical data to monitor the performance of surgical tasks, trigger events can be detected and responded to in real time, making it possible to change the level of automation during the current surgical procedure, such as by changing whether the monitored surgical task is automated and / or changing whether one or more of the subsequent surgical tasks of the surgical procedure are automated, which can increase the chances of a positive surgical outcome for the surgical procedure.

[0303] Following are numbered aspects of the disclosure that may or may not be claimed.

[0304] 1. A device for automating a surgical task, the device comprising:

[0305] A processor configured to:

[0306] receiving an indication of a surgical task to be performed using a surgical instrument, wherein capabilities of the surgical instrument are associated with a plurality of automation levels;

[0307] monitoring performance of the surgical task using the surgical instrument with the surgical instrument operating at a first level of automation among the plurality of levels of automation;

[0308] detecting a triggering event associated with said performance of said surgical task; and

[0309] Switching operation of the surgical instrument from the first automation level to a second automation level of the plurality of automation levels based on the triggering event.

[0310] 2. An apparatus according to aspect 1, wherein the capabilities of the surgical instrument include a set of surgical instrument tasks, and wherein the automation level is associated with automating one or more surgical instrument tasks in the set of surgical instrument tasks.

[0311] 3. An apparatus according to aspect 2, wherein automating one or more surgical tasks is associated with zero manual input from the surgeon.

[0312] 4. An apparatus according to aspect 2, wherein automating one or more surgical tasks is associated with reduced manual input from the surgeon.

[0313] 5. An apparatus according to aspect 2, wherein the execution of the surgical task is based on real-time surgical data.

[0314] 6. An apparatus according to aspect 5, wherein the real-time surgical data includes at least one of the following: user data, surgical environment data, surgical instrument data, task data, or historical data.

[0315] 7. The apparatus according to aspect 2, wherein the processor is further configured to:

[0316] detecting the triggering event by comparing the performance of the surgical task to a triggering event threshold; and

[0317] Operation of the surgical instrument is switched from the first automation level to the second automation level based on the triggering event, wherein the second automation level is associated with automating fewer surgical instrument tasks when compared to the first automation level.

[0318] 8. The apparatus of aspect 1, wherein the processor is further configured to:

[0319] detecting the triggering event by comparing the performance of the surgical task to a triggering event threshold; and

[0320] Operation of the surgical instrument is switched from the first automation level to the second automation level based on the triggering event, wherein the second automation level is associated with a higher manual input from a surgeon when compared to the first automation level.

[0321] 9. An apparatus according to aspect 6, wherein monitoring the performance of the surgical task includes comparing one or more of the real-time surgical data with corresponding ideal surgical data.

[0322] 10. A method for automating a surgical task, the method comprising:

[0323] receiving an indication of a surgical task to be performed using a surgical instrument, wherein capabilities of the surgical instrument are associated with a plurality of automation levels;

[0324] monitoring performance of the surgical task using the surgical instrument with the surgical instrument operating at a first level of automation among the plurality of levels of automation;

[0325] detecting a triggering event associated with said performance of said surgical task; and

[0326] Switching operation of the surgical instrument from the first automation level to a second automation level of the plurality of automation levels based on the triggering event.

[0327] 11. A method according to aspect 10, wherein the capabilities of the surgical instrument include a set of surgical instrument tasks, and wherein the automation level is associated with automating one or more surgical instrument tasks in the set of surgical instrument tasks.

[0328] 12. The method of aspect 11, wherein automating one or more surgical tasks is associated with zero manual input from the surgeon.

[0329] 13. The method of aspect 11, wherein automating one or more surgical tasks is associated with reduced manual input from the surgeon.

[0330] 14. The method of aspect 11, wherein the execution of the surgical task is based on real-time surgical data.

[0331] 15. The method of aspect 14, wherein the real-time surgical data comprises at least one of: user data, surgical environment data, surgical instrument data, task data, or historical data.

[0332] 16. The method according to aspect 11, further comprising:

[0333] detecting the triggering event by comparing the performance of the surgical task to a triggering event threshold; and

[0334] Operation of the surgical instrument is switched from the first automation level to the second automation level based on the triggering event, wherein the second automation level is associated with automating fewer surgical instrument tasks when compared to the first automation level.

[0335] 17. The method according to aspect 10, further comprising:

[0336] detecting the triggering event by comparing the performance of the surgical task to a triggering event threshold; and

[0337] Operation of the surgical instrument is switched from the first automation level to the second automation level based on the triggering event, wherein the second automation level is associated with a higher manual input from a surgeon when compared to the first automation level.

[0338] 18. The method of aspect 15, wherein monitoring the performance of the surgical task comprises comparing one or more of the real-time surgical data with corresponding ideal surgical data.

[0339] 19. An apparatus for automating a surgical task, the apparatus comprising:

[0340] A processor configured to:

[0341] operating at a first level of automation among a plurality of levels of automation associated with the surgical task;

[0342] obtaining an indication to switch to a second level of automation among the plurality of levels of automation associated with the surgical task; wherein the indication is based on detecting a trigger condition; and

[0343] Operating at a second level of automation among the plurality of levels of automation associated with the surgical task based on the indication.

[0344] 20. An apparatus according to aspect 19, wherein the indicating is based on monitoring execution of the first automation level.

[0345] 21. An apparatus according to aspect 20, wherein the execution of the first level of automation is based on real-time surgical data.

Claims

1. An apparatus for changing the automation level of a surgical instrument used to perform a surgical procedure, the apparatus comprising: A processor configured to: receiving an indication of a surgical procedure to be performed using a surgical instrument, the surgical procedure comprising a plurality of surgical tasks to be performed using the surgical instrument; monitoring performance of a surgical task from among the plurality of surgical tasks with the surgical instrument operating at a first level of automation; detecting a triggering event associated with said performance of said surgical task; as well as Switching operation of the surgical instrument from the first automation level to a second automation level based on the triggering event.

2. The device according to claim 1, wherein: The automation level corresponds to the number of automated surgical tasks among the plurality of surgical tasks.

3. The device according to claim 1 or 2, wherein: The second automation level is associated with automating fewer surgical tasks when compared to the first automation level.

4. The device according to claim 2 or 3, wherein: Automating surgical tasks is associated with zero manual input from the surgeon.

5. The device according to claim 2 or 3, wherein: Automating surgical tasks is associated with reduced manual input from the surgeon.

6. An apparatus according to any preceding claim, wherein: The performance of the surgical task is based on real-time surgical data.

7. The device according to claim 6, wherein: The real-time surgical data includes at least one of: user data, surgical environment data, surgical instrument data, and task data.

8. An apparatus according to any preceding claim, wherein: The processor is further configured to detect the trigger event by comparing the performance of the surgical task to a trigger event threshold.

9. The device according to claim 6, wherein: Monitoring the performance of the surgical task includes comparing one or more of the real-time surgical data with corresponding ideal surgical data, and detecting the occurrence of a trigger event when the comparison exceeds a trigger event threshold.

10. The device according to claim 9, wherein: The corresponding ideal surgical data is generated from the historical data of the surgical task.

11. An apparatus according to any preceding claim, wherein: The surgical procedure includes resection of tissue, and the plurality of surgical tasks include one or more of: controlling energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, reloading alignment slot management.

12. An apparatus according to any preceding claim, wherein: The surgical device includes a memory storing the plurality of surgical tasks associated with the surgical procedure and the surgical tasks to be automated for each autonomy level.

13. The device according to claim 12, wherein: The surgical instrument is suitable for performing two or more surgical procedures, the surgical instrument performing a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure, and wherein the surgical device includes a memory storing the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and wherein the memory also stores associations between the surgical tasks to be automated for each autonomy level of each surgical task in the first plurality of surgical tasks and the second plurality of surgical tasks.

14. An apparatus according to any preceding claim, wherein: The device is a surgical hub.

15. A computer-implemented method for providing commands or suggestions to change the automation level of a surgical instrument used to perform a surgical procedure, the method comprising: receiving an indication of a surgical procedure to be performed using a surgical instrument, wherein the surgical procedure includes a plurality of surgical tasks to be performed using the surgical instrument; monitoring performance of a surgical task among the plurality of surgical tasks with the surgical instrument operating at a first level of automation among a plurality of levels of automation; detecting a triggering event associated with said performance of said surgical task; and A command is provided to the surgical instrument or a suggestion is provided to a user to switch operation of the surgical instrument from the first automation level to a second automation level based on the triggering event.

16. The method according to claim 15, wherein: The automation level corresponds to the number of automated surgical tasks among the plurality of surgical tasks.

17. The method according to claim 16, wherein: The second automation level is associated with automating fewer surgical tasks when compared to the first automation level.

18. The method according to any one of claims 15 to 17, wherein: Monitoring the performance of the surgical task is based on real-time surgical data.

19. The method according to claim 18, wherein: The real-time surgical data includes at least one of: user data, surgical environment data, surgical instrument data, and task data.

20. The method according to claim 18 or 19, wherein: Monitoring the performance of the surgical task includes comparing one or more of the real-time surgical data with corresponding ideal surgical data, and detecting the occurrence of a trigger event when the comparison exceeds a threshold.

21. The method according to claim 20, wherein: The corresponding ideal surgical data is generated from the historical data of the surgical task.

22. The method according to claim 21, wherein: The method also includes generating the corresponding ideal surgical data.

23. The method according to any one of claims 15 to 22, wherein: The method includes retrieving from a memory the plurality of surgical tasks associated with the surgical procedure and the surgical tasks to be automated for each autonomy level.

24. The method according to claim 23, wherein: The surgical instrument is adapted to perform two or more surgical procedures, the surgical instrument performing a first plurality of surgical tasks for a first surgical procedure and a second plurality of surgical tasks for a second surgical procedure, and wherein the method includes retrieving from the memory indications of the first plurality of surgical tasks associated with the first surgical procedure and the second plurality of surgical tasks associated with the second surgical procedure, and retrieving from the memory associations between the surgical tasks to be automated for each autonomy level of each surgical task in the first plurality of surgical tasks and the second plurality of surgical tasks.

25. A computer program comprising instructions which, when a computer executes the program, cause the computer to perform the method according to any one of claims 15 to 24.

26. A computer readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 15 to 24.

27. An apparatus for performing multiple surgical tasks during a surgical procedure, the apparatus comprising: A processor configured to: Operate at the first automation level; Obtaining an indication to switch to a second automation level; wherein the indication is based on detecting a trigger condition; as well as Operating at the second automation level based on the indication.

28. The device according to claim 27, wherein The first automation level and the second automation level differ in the number of automated surgical tasks of the surgical procedure.

29. The device according to claim 27 or 28, wherein The indicating is based on monitoring the performance of a surgical task among the plurality of surgical tasks at the first automation level.

30. The device according to claim 29, wherein: The performance of the surgical task at the first automation level is based on real-time surgical data.

Images