Detection of failure reduction associated with an autonomous surgical task

Abstract

Systems, methods, and means for detecting failure mitigation associated with a surgical task are described herein. The device can perform a first autonomous function associated with the surgical task. Performing the first autonomous function can be associated with control feedback. The first autonomous function can be monitored by tracking one or more outputs associated with the control feedback. If one or more outputs exceed a failure threshold, the device may switch from performing the first autonomous function to performing a second autonomous function associated with the surgical task. The second autonomous function can be associated with failure mitigation feedback.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application is related to the following applications filed simultaneously, the content of each of which is incorporated herein by reference. ·U.S. Patent Application entitled "METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM" filed together with this specification and having Attorney Docket No. END9430USNP1. ·U.S. Patent Application entitled "DYNAMICALLY DETERMINING SURGICAL AUTONOMY LEVEL" filed together with this specification and having Attorney Docket No. END9430USNP2.

Background Art

[0002] Surgical operations are typically performed in an operating room or room within a medical facility, such as a hospital. Various surgical devices and systems are utilized in the performance of surgical operations. In the digital information age, medical systems and facilities often implement systems or procedures using newer and improved technologies more slowly due to the general desire to maintain patient safety and conventional practices.

Summary of the Invention

Means for Solving the Problems

[0003] Systems, methods, and means for detecting failure mitigation associated with a surgical task are described herein. A device can perform a first autonomous function associated with a surgical task. Performing the first autonomous function can be associated with control feedback. The first autonomous function can be monitored by tracking one or more outputs associated with the control feedback. If one or more of the outputs exceed a failure threshold, the device may switch from performing the first autonomous function to performing a second autonomous function associated with the surgical task. The second autonomous function can be associated with failure mitigation feedback. The device may generate a magnitude of a failure based on one or more of the outputs that exceeded the failure threshold. The device can adjust one or more parameters associated with the failure mitigation feedback based on the magnitude of the failure. In an example, the magnitude of the failure can be generated based on one or more of a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs that exceeded the failure threshold, or a degree to which one or more of the outputs exceeded the failure threshold.

[0004] A device for controlling the execution of a surgical task by a surgical instrument is described. The device comprises a processor. The processor is configured to monitor a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task. The processor is configured to control the surgical instrument according to a first model based on the one or more tracked outputs. The processor is configured to switch the surgical instrument from the execution of the first autonomous function to the execution of a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model when at least one of the one or more tracked outputs exceeds a fault threshold. Advantageously, these functions enable the combination of the device and the surgical instrument to function as a self - contained and fault - tolerant autonomous surgical system. Many instrument controllers that use any kind of output or feedback to control the instrument provide control instructions as a simple function of the tracked output(s) (e.g., setting parameters as a linear function of the output value for the purpose of keeping the output constant, etc.). Thus, when a very abnormal or unexpected output is detected (typically associated with some kind of failure), the control instructions provided to the instrument based on the output are also abnormal and are likely to be inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds the fault threshold. The device may be external to the surgical instrument or may be a processor or control circuit included within the surgical instrument.

[0005] The first autonomous function and the second autonomous function may be different autonomous functions. This can be beneficial because it allows for a change in function when a fault is detected, in order to enable the procedure to be safely paused, stopped, interrupted, or completed. For example, the first autonomous function may include a clamping operation, and when the threshold force is exceeded, the second autonomous function may be a jaw - opening operation.

[0006] Controlling a surgical instrument according to a first model may include providing control feedback generated by the first model to the surgical instrument. The control feedback may be associated with setting and / or updating one or more parameters of the surgical instrument. One or more parameters of the surgical instrument may include parameters of a script for controlling the surgical instrument.

[0007] The control feedback may be configured to be used as an input to one or more actuators of the surgical instrument. The control feedback may be configured to set parameters via a message to the surgical instrument.

[0008] One or more outputs associated with a first autonomous function and / or a surgical task may include measured real-world data derived from one or more of the measurements of sensors on the surgical instrument, measurements of wearable sensors that measure biomarkers or other health markers of a patient or surgeon associated with the surgical task, optionally including heart rate, and measurements of the positions of one or more actuators disposed on the surgical instrument performing the surgical task.

[0009] Based on the tracked output, controlling the surgical instrument according to the first model may include controlling the surgical instrument according to at least one output, or comparing at least one output with an ideal output generated by the first model to determine the difference between the two and controlling the surgical instrument according to the difference.

[0010] Determining whether at least one of one or more outputs has exceeded a failure threshold may include determining that at least one output includes measured real-world data outside a given range, or comparing at least one output including the measured real-world data with an ideal output generated by the first model to determine the difference between the two and determining that the difference is outside a given range.

[0011] Determining whether at least one of one or more outputs has exceeded a fault threshold can include determining whether a predetermined number of each of the one or more outputs has exceeded the fault threshold. The predetermined number may be manually input by a user of the surgical instrument.

[0012] Controlling a surgical instrument according to a fault mitigation model can include generating or setting one or more parameters for the surgical instrument, sending a message to a user of the surgical instrument, reducing the level of autonomy of the surgical instrument, and / or switching the surgical instrument from an autonomous setting to a manual setting, ending all operations associated with the surgical task, or reducing system performance to enable a safe end to the surgical task, where it is known or expected that doing so reduces the risks associated with completion of the surgical task. Switching the surgical instrument from execution of a first autonomous function to execution of a second autonomous function associated with the surgical task can correspond to a switch to less autonomous control, a switch of the operating parameters of the device (e.g., reducing the speed of a cutter, reversing the cutter, etc.), a switch to an algorithm that halts operation (e.g., switching from clamping to releasing), or a change in the power level of an energy device. Advantageously, each of these fault mitigation responses can provide a response to a fault detected during execution of the surgical task, which can reduce the risk of harm to the patient. For example, ending all operations associated with the surgical task can stop an incorrect autonomous function within its track, preventing it from inadvertently doing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation means (that do not simply end the operation) can provide benefits over a simple fault mitigation control that simply halts the process within its track. This is because in various surgical contexts, better clinical outcomes (e.g., reduced harm to the patient or reduced complications) can be achieved by continuing the operation of the instrument in a safety-oriented manner rather than by ending its operation. For example, a surgical instrument having an end effector for clamping tissue, where the clamp arm is biased to a closed position, can be made safe by the controller by setting the actuator to unclamp the jaws of the end effector and thus release any tissue held therein.

[0013] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamp jaws of the end effector. During “normal” operation (i.e., operation according to the first model), this clamp force feedback may be used to control the speed of an actuator configured to advance the knife member and / or thread assembly. This can include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model can direct adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a high force is detected (indicating a larger, more robust tissue type) and slightly decreasing the firing speed when a low force is detected (indicating a smaller, more delicate tissue type). However, if it is determined that the force on the jaws has dropped below a failure threshold (e.g., has dropped to or near zero), this may indicate that the tissue has slipped out of the jaws of the end effector. Accordingly, the system can switch to a failure mitigation model, which can direct setting the firing speed to a negative value (corresponding to reversing the direction of firing and driving the knife in the opposite direction within its recessed housing), which is known (or at least reasonably foreseeable) to make the instrument safe by allowing repositioning of the end effector without significant risk to the patient or operator. The system can optionally also warn the user of the fact that this is occurring.

[0014] As another non-limiting example, when an unexpectedly strong force is detected to close the end effector of a surgical instrument, the system may switch to a failure mitigation model that can ramp up or ramp down the motor of the surgical instrument.

[0015] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply high-frequency (RF) and ultrasonic energy to tissue can be controlled according to a first model while the impedance level at the end effector electrode is being monitored. The first model may apply RF energy to the tissue and may adjust the energy supply parameters (e.g., of the generator) based on the monitored impedance to ensure that the energy supply (e.g., power) at the surgical site remains substantially constant. However, if an unexpected spike in impedance (either very high or very low) is detected, this can exceed a fault threshold (defined as the range of impedance within which the detected impedance is expected to decrease during normal operation of the device). Thus, a switch can be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example can include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent unwanted cutting of the tissue.

[0016] One or more of the tracked outputs can be the first tracked one or more outputs. The processor can be further configured to monitor a second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task. The processor can be further configured to control the surgical instrument according to a fault mitigation model that can be based on the second tracked one or more outputs.

[0017] The processor can be further configured to generate a magnitude of the fault based on the one or more outputs if the one or more outputs exceed a fault threshold. Controlling the surgical instrument according to a fault mitigation model can include adjusting one or more parameters of the surgical instrument based on the magnitude of the fault. Advantageously, this intelligently adjusts the response of the device and instrument to the fault to be proportional to the magnitude of that fault.

[0018] The magnitude of the failure can be generated based on at least one of a surgical context, a first type of autonomous function, a second type of autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or the degree to which one or more outputs exceed the failure threshold. Advantageously, this enables, for example, the detection of failures that may be significant but not numerous, and vice versa, allowing a balance to be taken between importance and quantity when the device reduces failures. Additionally or alternatively, it advantageously enables more important surgical functions (e.g., life and death) to be prioritized over less important surgical functions (e.g., utility / ease of use for the surgeon) for failure reduction.

[0019] The processor may be further configured to perform an analysis on one or more tracked outputs. The processor may be further configured to generate a comparison output based on the analysis. The processor may be further configured to perform a switch from the execution of the first autonomous function to the execution of the second autonomous function if the comparison output exceeds a failure threshold. Advantageously, in this way, the switching of the autonomous function can be performed in response to a relative output problem rather than an absolute output problem (e.g., a comparison can be made with an ideal, desired, and / or conceptual level of the output(s)). Additionally or alternatively, it may be beneficial to make this function switch responsive to the relationship between one or more outputs, allowing for more nuances.

[0020] The processor may be further configured to assign a weight to each of the one or more tracked outputs based on at least one of past data, a surgical context, a first type of autonomous function, a second type of autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or the degree to which one or more outputs exceed the failure threshold.

[0021] The processor may be further configured to send an indication of a fault message to a user of a surgical instrument associated with a surgical task when one or more outputs exceed a fault threshold. The fault message may include a fault type and a magnitude of the fault.

[0022] The fault message may comprise a set of recommendations based on a type of a first autonomous function, a type of a second autonomous function, past data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or a degree to which one or more outputs exceed the fault threshold.

[0023] The processor may be further configured to determine a fault threshold and fault mitigation feedback based on training data.

[0024] The training data may be based on at least one of past data, a surgical context, a type of a first autonomous function, a type of a second autonomous function, past data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or a degree to which one or more outputs exceed the fault threshold.

[0025] The fault threshold may be adjusted based on situation recognition.

[0026] A computer-implemented method for controlling the execution of a surgical task by a surgical instrument is described. The method includes monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task. The method includes providing a digital command or user recommendation for controlling the surgical instrument according to a first model based on the one or more tracked outputs. The method further includes providing a digital command or user recommendation for switching the surgical instrument from the execution of the first autonomous function to the execution of a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model according to a determination that at least one of the one or more outputs has exceeded a fault threshold. Advantageously, these functions enable the combination of the computer implementing the method and the surgical instrument to function as a self-contained and fault-tolerant autonomous surgical system. Many instrument control methods that use any kind of output or feedback to control the instrument involve providing control instructions as a simple function of the tracked output(s) (e.g., setting parameters as a linear function of the output value for the purpose of keeping the output constant). Thus, when a very abnormal or unexpected output is detected (typically associated with some kind of fault), the control instructions provided to the instrument based on the output are also abnormal and are likely to be inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds the fault threshold. The computer implementing the method may be external to the surgical instrument, or may be a processor or control circuit included within the surgical instrument itself.

[0027] The first autonomous function and the second autonomous function may be different autonomous functions. This can be beneficial because it allows for a change in function when a fault is detected, which enables the procedure to be safely paused, stopped, interrupted, or completed. For example, the first automated function may include a clamping operation, and when a threshold force is exceeded, the second autonomous function may be a jaw-opening operation.

[0028] Digital commands or user recommendations for controlling a surgical instrument according to a first model may include one or more values for setting or updating parameters of the surgical instrument. One or more parameters of the surgical instrument may include parameters of a script for controlling the surgical instrument.

[0029] The values may be suitable for use as inputs for one or more actuators of the surgical instrument. The digital command may be a message to the surgical instrument configured to set a parameter of the surgical instrument.

[0030] One or more outputs associated with a first autonomous function and / or a surgical task may include measured real-world data derived from one or more of the measurements of sensors on the surgical instrument. The output may further include measurements of a wearable sensor that measures a patient's or surgeon's biomarker or other health marker associated with the surgical task, optionally including a heart rate. The output may include measurements of the position of one or more actuators disposed on the surgical instrument performing the surgical task.

[0031] Digital commands or user recommendations for controlling a surgical instrument according to a first model based on the tracked output may be either digital commands or user recommendations for controlling the surgical instrument according to at least one output, or digital commands or user recommendations for controlling the surgical instrument according to a difference determined by comparing at least one output with an ideal output generated by the first model.

[0032] Determining whether at least one of the one or more outputs has exceeded a fault threshold may include determining that at least one output includes measured real-world data outside a given range, or comparing at least one output including measured real-world data to an ideal output generated by a first model to determine a difference between the two, and determining that the difference is outside a given range.

[0033] Determining whether at least one of the one or more outputs has exceeded a fault threshold may include determining whether a predetermined number of the one or more outputs each exceed the fault threshold. The predetermined number may be manually input by a user of the surgical instrument.

[0034] Digital commands or user recommendations for switching a surgical instrument from performing a first autonomous function to performing a second autonomous function may include generating or setting one or more parameters for the surgical instrument that are known or expected to reduce the risk associated with the completion of a surgical task, sending a message to the user of the surgical instrument, reducing the level of autonomy of the surgical instrument, and / or switching the surgical instrument from an autonomous setting to a manual setting, ending all operations associated with the surgical task, or reducing system performance to enable a safe end to the surgical task. Switching a surgical instrument from performing a first autonomous function to performing a second autonomous function associated with a surgical task may correspond to switching to less autonomous control, switching the operating parameters of the device (e.g., reducing the speed of a cutter, reversing a cutter, etc.), switching to an algorithm that stops an operation (e.g., switching from clamping to releasing), or changing the power level of an energy device. Advantageously, each of these fault mitigation responses can provide a response to a fault detected during the execution of a surgical task, which can reduce the risk of harm to the patient. For example, ending all operations associated with the surgical task can stop an incorrect autonomous function within its track, preventing it from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation means described above (which do not simply end the operation) can provide benefits over a simple fault mitigation control that simply stops the process within its track. This is because in various surgical contexts, better clinical outcomes (e.g., reduced harm to the patient or reduced complications) can be achieved by continuing the operation of the instrument in a safety-oriented manner rather than by ending its operation. For example, a surgical instrument having an end effector for clamping tissue, where the clamp arm is biased to a closed position, can be made safe by the controller by setting the actuator to unclamp the jaws of the end effector and thus release any tissue held therein.

[0035] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamp jaws of the end effector. During “normal” operation (i.e., operation according to the first model), this clamp force feedback can be used to control the speed of an actuator configured to advance the knife member and / or the thread assembly. This can include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model can direct adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a high force is detected (indicating a larger, more robust tissue type) and slightly decreasing the firing speed when a low force is detected (indicating a smaller, more fragile tissue type). However, if it is determined that the force on the jaws has dropped below a failure threshold (e.g., has dropped completely to, or nearly to, zero), this may indicate that the tissue has slipped out of the jaws of the end effector. Thus, the system can switch to a failure mitigation model, which can direct setting the firing speed to a negative value (corresponding to reversing the direction of firing and driving the knife in the opposite direction within its recessed housing), as this is known (or at least reasonably foreseeable) to make the instrument safe by allowing repositioning of the end effector without significant risk to the patient or operator. The system can optionally also warn the user of the fact that this is occurring.

[0036] As another non-limiting example, when an unexpectedly strong force is detected to close the end effector of a surgical instrument, the system may switch to a failure mitigation model that can ramp up or ramp down the motor of the surgical instrument.

[0037] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply high-frequency (RF) and ultrasonic energy to tissue can be controlled according to a first model while the impedance level at the end effector electrode is being monitored. The first model may apply RF energy to the tissue and adjust the energy supply parameters (e.g., of the generator) based on the monitored impedance to ensure that the energy supply (e.g., power) at the surgical site remains substantially constant. However, if an unexpected spike in impedance (either very high or very low) is detected, this can exceed a fault threshold (defined as the range of impedance within which the detected impedance is expected to decrease during normal operation of the device). Thus, a switch can be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example can include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent unwanted cutting of the tissue.

[0038] One or more of the tracked outputs can be the first tracked one or more outputs. The method may further include monitoring a second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task. Digital commands or user recommendations for controlling the surgical instrument according to the fault mitigation model can be based on the second tracked one or more outputs.

[0039] The method may further include generating a magnitude of the fault based on the one or more outputs if the one or more outputs exceed a fault threshold. Digital commands or user recommendations can be for adjusting one or more parameters of the surgical instrument based on the magnitude of the fault. Advantageously, this intelligently adjusts the response of the computer and the instrument to the fault to be proportional to the magnitude of that fault.

[0040] The magnitude of the fault can be generated based on at least one of the surgical context, the type of the first autonomous function, the type of the second autonomous function, the past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold. Advantageously, this enables, for example, faults that may be significant but not numerous to be detected, and vice versa, and enables a balance to be struck between importance and quantity when the computer reduces the fault. Additionally or alternatively, it advantageously enables more critical surgical functions (e.g., life and death) to be prioritized over less critical surgical functions (e.g., utility / ease of use for the surgeon) for fault reduction.

[0041] The method may further include performing an analysis on one or more of the tracked outputs. The method may further include generating a comparison output based on the analysis. A digital command or user recommendation for switching the surgical instrument from the execution of the first autonomous function to the execution of the second autonomous function may be provided according to a determination that the comparison output exceeds the fault threshold. Advantageously, in this way, the switching of the autonomous function can be performed in response to a relative output problem rather than an absolute output problem (e.g., a comparison can be made with an ideal, desired, and / or conceptual level of the output(s)). Additionally or alternatively, it may be beneficial to enable more nuances by making the switching of this function responsive to the relationship between one or more outputs.

[0042] The method may further include assigning a weight to each of the one or more tracked outputs based on at least one of past data, the surgical context, the type of the first autonomous function, the type of the second autonomous function, the past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold.

[0043] The method may further include transmitting a fault message to a user of a surgical instrument associated with the surgical task if the one or more outputs exceed a fault threshold. The fault message may include a fault type and a fault magnitude.

[0044] The fault message may comprise a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceeded the fault threshold, or the extent to which one or more outputs exceeded the fault threshold.

[0045] The method may further include determining a fault threshold and a fault mitigation feedback based on the training data.

[0046] The training data may be based on at least one of historical data, a surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceeded a fault threshold, or the extent to which one or more outputs exceeded a fault threshold.

[0047] Fault thresholds may be adjusted based on situational awareness.

[0048] A computer program product is described for causing a computer to carry out any of the above methods.

[0049] A non-transitory computer readable storage medium is described that contains computer readable instructions that, when executed by a computer, cause the computer to perform any of the methods described above. [Brief description of the drawings]

[0050]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Best Mode for Carrying Out the Invention

[0051] FIG. 1 is a block diagram of a computer-implemented surgical system 20000. An exemplary surgical system such as surgical system 20000 can include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, each surgical system 20002 can include a computer-implemented bidirectional surgical system. For example, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 that communicate with a cloud computing system 20008, as described, for example, in FIG. 2. The cloud computing system 20008 can include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Exemplary surgical systems 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include, for example, one or more devices used to measure one or more environmental attributes, as further described in FIG. 2. The robotic system 20013 may include, for example, a plurality of devices used to perform a surgical procedure, as further described in FIG. 2.

[0052] The surgical system 20002 can communicate with a remote server 20009 that can be part of a cloud computing system 20008. In one example, the surgical system 20002 may communicate with the remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, the patient sensing system may communicate directly with the remote server 20009. The surgical system 20002 and / or its components may use one or more of the cellular protocols of GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A), new radio (NR) or 5G to communicate with the remote server 20009 via a cellular transmission / reception point (TRP) or base station.

[0053] The surgical hub 20006 can have a cooperative interaction with one of more means for displaying images from a laparoscope, as well as information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from one or more sensing systems 20011 and send notification or control messages to one or more sensing systems 20011. The surgical hub 20006 can transmit and / or receive information, including notification information, to and from a human interface system 20012. The human interface system 20012 can include one or more human interface devices (HIDs). The surgical hub 20006 can transmit and / or receive acoustic devices, notification or control information to a display, and / or control information to various devices that communicate with the surgical hub.

[0054] For example, as discussed in FIG. 1, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and one or more patient sensing systems), and an environmental sensing system 20015. One or more sensing systems 20001 may measure data regarding various biomarkers. One or more sensing systems 20001 may use one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc., to measure biomarkers. One or more sensors may use one of more of the sensing techniques such as photoplethysmography, electrocardiogram examination, electroencephalogram examination, colorimetric analysis, impedancemetry, potential difference measurement, current measurement, etc., to measure biomarkers as described herein.

[0055] Biomarkers measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood potential of hydrogen, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous tension, circadian rhythm, and / or menstrual cycle.

[0056] Biomarkers can be related to physiological systems, including but not limited to the behavioral and psychological, cardiovascular, renal, skin, nervous, gastrointestinal, respiratory, endocrine, immune, tumor, musculoskeletal, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgical system 20000. Information from biomarkers may be determined and / or used by the computer-implemented patient and surgical system 20000 to, for example, improve the above systems and / or improve patient outcomes. One or more sensing systems 20001, biomarkers 20005, and physiological systems are described in more detail in U.S. Patent Application No. 17 / 156,287, filed January 22, 2021, entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS" (Attorney Docket No. END9290USNP1), the disclosure of which is incorporated herein by reference in its entirety.

[0057] Figure 2 shows an example of a surgical system 20002 in an operating room. As illustrated in Figure 2, a patient is operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more HCP sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding the HCPs may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors deployed in the operating room. The HCP sensing systems 20020 and the environmental sensing systems communicate with the surgical hub 20006 and may further communicate with one or more cloud servers 20009 of the cloud computing system 20008, as shown in Figure 1. The environmental sensing system may be used to measure one or more environmental attributes, such as the position of the HCPs in the operating room, the movement of the HCPs, the ambient noise in the operating room, the temperature / humidity in the operating room, etc.

[0058] As illustrated in FIG. 2, the main display 20023 and one or more audio output devices (e.g., speaker 20019) are placed within the sterile field so as to be visible to the operator on the operating table 20024. Additionally, the visualization / notification tower 20026 is placed outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human interface device (HID) 20027 and a second non-sterile HID 20029 facing opposite each other. The HID may be a display, or may be a display having a touch screen that enables a human to directly interface with the HID. The human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to regulate the information flow to the operators inside and outside the sterile field. In one example, the surgical hub 20006 may cause the HID (e.g., the main HID 20023) to display notifications and / or information regarding the patient and / or surgical steps. In one example, the surgical hub 20006 may prompt and / or receive input from a person within the sterile field or non-sterile area. In one example, the surgical hub 20006 may cause the HID to display a snapshot of the surgical site recorded by the imaging device 20030 on the non-sterile HID 20027 or 20029 while maintaining a live video of the surgical site on the main HID 20023. The snapshot on the non-sterile display 20027 or 20029 may, for example, permit a non-sterile operator to perform diagnostic steps related to the surgery.

[0059] In one aspect, the surgical hub 20006 may be configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to the main display 20023 within the sterile field, so that it can be viewed by the sterile operator on the operating table. In one example, the input may be in the form of a modification to a snapshot displayed on the non-sterile display 20027 or 20029, which may be sent to the main display 20023 by the surgical hub 20006.

[0060] Referring to FIG. 2, the surgical instrument 20031 is used as part of a surgical system 20002 in a surgical operation. The hub 20006 can also be configured to regulate the information flow to the display of the surgical instrument 20031. For example, in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 20026 are sent by the hub 20006 to the surgical instrument display within the sterile field, where they can be viewed by the operator of the surgical instrument 20031. Exemplary surgical instruments suitable for use with the surgical system 20002 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0061] FIG. 2 illustrates an example of a surgical system 20002 used to perform surgery on a patient lying on an operating table 20024 within an operating room 20035. A robotic system 20034 can be used as part of the surgical system 20002 in a surgical operation. The robotic system 20034 can include a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. While the surgeon views the surgical site through the surgeon's console 20036, the patient-side cart 20032 can operate at least one removably coupled surgical tool 20037 through a minimally invasive incision in the patient's body. An image of the surgical site can be acquired by a medical imaging device 20030 that can be operated by the patient-side cart 20032 to change the orientation of the imaging device 20030. The robotic hub 20033 can be used to process the image of the surgical site and then display it to the surgeon through the surgeon's console 20036.

[0062] Other types of robotic systems can be readily adapted for use with the surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019 / 0201137 (A1) (U.S. Patent Application No. 16 / 209,407) entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0063] Various examples of cloud-based analysis methods implemented by a cloud computing system 20008 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569 (A1) (U.S. Patent Application No. 16 / 209,403) entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB", filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0064] In various embodiments, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0065] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the operative field. The one or more image sensors may be capable of receiving light reflected or refracted from the operative field, including light reflected or refracted from tissue and / or surgical instruments.

[0066] The one or more illumination sources may be configured to emit electromagnetic energy within the visible spectrum as well as the invisible spectrum. The visible spectrum is, in some cases, also referred to as the optical spectrum or emission spectrum and is a portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye) and may be referred to as visible light or simply light. Typically, the human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0067] The invisible spectrum (e.g., non-emission spectrum) is a portion of the electromagnetic spectrum that is located below and above the visible spectrum (i.e., wavelengths less than approximately 380 nm and greater than approximately 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet light, x-rays, and gamma ray electromagnetic radiation.

[0068] In various aspects, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0069] The imaging device may employ multispectral monitoring to distinguish topography from the underlying structure. A multispectral image captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelengths can be separated by a filter or by using an instrument with sensitivity to specific wavelengths including frequencies beyond the visible light range, e.g., IR, and light from ultraviolet. Spectral imaging enables extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is detailed under the heading “Advanced Imaging Acquisition Module” in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after a surgical task for performing one or more of the above-described tests on the treated tissue has been completed. It is understood that strict sterilization of the operating room and surgical instruments is required during any surgical procedure. The strict hygiene and sterilization conditions required in the “operating room,” i.e., the operating or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize anything that comes into contact with the patient or enters the sterile field, including the imaging device 20030 and its accessories and components. It will be understood that the sterile field can be considered a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding a patient prepared for surgery. The sterile field can include properly attired and scrubbed team members, as well as all equipment and fixtures within that area.

[0070] The wearable sensing system 20011 shown in FIG. 1 may include one or more sensing systems, such as the HCP sensing system 20020 shown in FIG. 2. The HCP sensing system 20020 may include a sensing system for monitoring and detecting a set of physical and / or physiological states of a healthcare provider (HCP). The HCP may generally be one or more healthcare providers who assist a surgeon or other healthcare service provider. In one example, the sensing system 20020 may measure a set of biomarkers to monitor the heart rate of the HCP. In one example, the sensing system 20020 (e.g., a watch or a wristband) worn on the wrist of a surgeon may use an accelerometer to detect hand movement and / or shake, and determine the magnitude and frequency of tremors. The sensing system 20020 may transmit measurement data associated with a set of biomarkers and data associated with the physical state of the surgeon to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the position of the HCP's hand / body. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include devices such as a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the operating room. The surgical hub 20006 may, alone or in communication with a cloud computing system, use the surgeon biomarker measurement data and / or environmental sensing information to, for example, modify the control algorithm of a handheld instrument or the average latency of a robotic interface to minimize tremors. In one example, the HCP sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit measurement data associated with the surgeon biomarkers to the surgical hub 20006.The HCP sensing system 20020 may use one or more of the RF protocols of Bluetooth (registered trademark), Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 low-power wireless personal area network (6LoWPAN), and Wi-Fi to communicate with the surgical hub 20006. The surgeon biomarkers may include one or more of stress, heart rate, etc. The environmental measurements from the operating room may include ambient noise levels related to the movement of the surgeon or patient, surgeon and / or staff, the attention level of the surgeon and / or staff, etc.

[0071] The surgical hub 20006 may adaptively control one or more surgical instruments 20031 using the surgeon biomarker measurement data associated with the HCP. For example, the surgical hub 20006 may send a control program to the surgical instrument 20031 to control its actuator to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may send the control program based on situation awareness and / or circumstances regarding the importance or criticality of the task. The control program may instruct the instrument to change its operation to provide more control when control is needed.

[0072] Figure 3 shows an exemplary surgical system 20002 having a surgical hub 20006. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014 via a modular control unit. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain embodiments, as illustrated in Figure 3, the hub 20006 further includes an exhaust smoke module 20054 and / or a suction / irrigation module 20055. The various modules and systems can be connected directly or via the communication module 20056 to the modular control unit via a router. Operating room devices can be coupled to cloud computing resources and data storage via the modular control unit. The human interface system 20012 can include a display subsystem and a notification subsystem.

[0073] The modular control unit may be connected to a non-contact sensor module. The non-contact sensor module may use ultrasonic, laser type, and / or similar non-contact measurement devices to measure the dimensions of the operating room and generate a map of the operating room. Other distance sensors can be used to determine the boundaries of the operating room. In U.S. Provisional Patent Application No. 62 / 611,341, filed on December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," which is hereby incorporated by reference in its entirety, as described under the heading "Surgical Hub Spatial Awareness Within an Operating Room" in the same document, an ultrasonic-based non-contact sensor module can scan the operating room by transmitting ultrasonic bursts and receiving the echoes when the ultrasonic bursts are reflected from the outer wall of the operating room. The sensor module may be configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. A laser-based non-contact sensor module can scan the operating room, for example, by transmitting laser light pulses, receive the laser light pulses reflected from the outer wall of the operating room, compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room, and adjust the Bluetooth pairing distance limit.

[0074] During surgery, applying energy to tissue for sealing and / or cutting is generally associated with smoke evacuation, aspiration of excess fluid, and / or perfusion of tissue. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgery. Valuable time can be lost in dealing with this problem during surgery. To untangle the lines, it may be necessary to remove the lines from their corresponding modules, which may require resetting the modules. The hub module type enclosure 20060 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglements. Aspects of the present disclosure present a surgical hub 20006 for use in surgeries involving the application of energy to tissue at the surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combined generator module slidably receivable within the docking station of the hub enclosure 20060. The docking station includes data contacts and power contacts. The combined generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit. In one aspect, the combined generator module also includes a smoke evacuation component, at least one energy supply cable for connecting the combined generator module to a surgical instrument, at least one smoke evacuation component configured to discharge smoke, fluid, and / or particulates generated by the application of therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke evacuation component. In one aspect, the fluid line may be a first fluid line, and a second fluid line may extend from a remote surgical site to a suction and perfusion module 20055 slidably receivable within the hub enclosure 20060. In one aspect, the hub enclosure 20060 may include a fluid interface. Certain surgeries may require applying two or more energy types to tissue.One type of energy may be more beneficial for cutting tissue, while a different type of energy may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut the sealed tissue. Aspects of the present disclosure present a solution where a hub-module enclosure 20060 is configured to house different generators and facilitate two-way communication between them. One advantage of the hub-module enclosure 20060 is that it allows for the quick removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port that includes first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is also slidably movable to disengage from the electrical engagement with the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy different from the first energy for application to tissue, and a second docking station having a second docking port that includes second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is also slidably movable to disengage from the electrical engagement with the second power and data contacts. Additionally, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port configured to facilitate communication between the first energy generator module and the second energy generator module.Referring to FIG. 3, aspects of the present disclosure are presented regarding a hub module type enclosure 20060 that enables modular integration of a generator module 20050, a smoke exhaust module 20054, and a suction / irrigation module 20055. The hub module type enclosure 20060 further facilitates two-way communication between module 20059, module 20054, and module 20055. The generator module 20050 may include integrated monopolar components, bipolar components, and ultrasonic components supported within a single housing unit slidably insertable into the hub's module type enclosure 20060. The generator module 20050 may be configured to connect to a monopolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the hub module type enclosure 20060. The hub module type enclosure 20060 can be configured to facilitate the insertion of multiple generators and two-way communication between the generators docked to the hub module type enclosure 20060 such that the multiple generators function as a single generator.

[0075] FIG. 4 illustrates a surgical data network having a set of communication hubs configured to connect to a cloud, a set of sensing systems, environmental sensing systems, and a set of other modular devices disposed in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specially equipped for surgery within a medical facility, according to at least one aspect of the present disclosure.

[0076] As illustrated in FIG. 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices disposed within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 that may include a remote server 20067 connected to a remote storage 20068). The modular communication hub 20065 and the devices may be connected in a room within a medical facility specially equipped for surgical procedures. In one aspect, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicate with a network router 20066. The modular communication hub 20065 may also be coupled to a local computer system 20063 to provide local computer processing and data manipulation.

[0077] The computer system 20063 may include a processor and a network interface 20100. The processor may be coupled via a system bus to a communication module, storage, memory, non-volatile memory, and an input / output (I / O) interface. The system bus can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, using any of various available bus architectures, examples of which include a 9-bit bus, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus, but are not limited thereto.

[0078] The processor may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex by Texas Instruments. In one aspect, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core has on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum of 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle serial random access memory (SRAM), an internal read-only memory (ROM) with StellarisWare® software, 2KB of electrically erasable programmable read-only memory (EEPROM) and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, the details of which are available in the product datasheet.

[0079] In one example, the processor may include a safety controller with two controller-based families such as TMS570 and RM4x, also known by the trade name Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, while providing scalable performance, connectivity, and memory options, and providing a high level of integrated safety mechanisms.

[0080] It should be understood that the computer system 20063 may include software that functions as a medium between the described user and the basic computer resources in a suitable operating environment. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may utilize the resource management by the operating system via program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0081] A user can input commands or information into the computer system 20063 via an input device connected to the I / O interface. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styli, touch pads, keyboards, microphones, joysticks, game pads, satellite broadcast receiving antennas, scanners, TV tuner cards, digital cameras, digital video cameras, webcams, etc. These and other input devices are connected to the processor 20102 through the system bus via an interface port. Examples of interface ports include serial ports, parallel ports, game ports, and USB. Output devices use some of the same type of ports as input devices. Thus, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to an output device. Output adapters may be provided, among other output devices that may require special adapters, to illustrate that several output devices such as monitors, displays, speakers, and printers can exist. Examples of output adapters include video and sound cards that provide connection means between the output device and the system bus, but this is for illustration purposes only and is not limiting. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0082] The computer system 20063 can operate in a networked environment using logical connections to one or more remote computers, such as a cloud computer, or a local computer. The remote cloud computer can be, for example, a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based device, a peer device, or other common network nodes, but typically includes many or all of the elements described with respect to the computer system. For the sake of brevity, only a memory storage device is illustrated together with the remote computer. The remote computer can be logically connected to the computer system via a network interface and subsequently physically connected via a communication connection. The network interface can include communication networks such as a local area network (LAN) and a wide area network (WAN). Examples of LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE802.3, Token Ring / IEEE802.5, and the like. Examples of WAN technologies include circuit-switched networks such as point-to-point links, Integrated Services Digital Network (ISDN) and its variations, packet-switched networks, and Digital Subscriber Line (DSL), but are not limited thereto.

[0083] In various examples, computer system 20063 may include an image processor, an image processing engine, a media processor, or any special digital signal processor (DSP) used for processing digital images. The image processor can enhance speed and efficiency using parallel computing with single instruction, multiple data (SIMD), or multiple instruction, multiple data (MIMD) techniques. The digital image processing engine can perform various tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0084] The communication connection part may refer to the hardware / software used to connect a network interface to a bus. For the sake of exemplary clarity, the communication connection part is shown inside computer system 20063, but the communication connection part may be outside computer system 20063. For illustrative purposes only, the hardware / software required for connection to a network interface can include modems such as ordinary telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and internal and external technologies such as Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0085] The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switching. A passive surgical data network functions as a conduit for data, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through the surgical data network being monitored and includes additional features that configure each port within network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0086] The modular devices 1a - 1n arranged in the operating room can be connected to the modular communication hub 20065. The network hub 20061 and / or the network switch 20062 can be connected to the network router 20066 to connect the devices 1a - 1n to the cloud computing system 20064 or the local computer system 20063. The data associated with the devices 1a - 1n may be transferred via the router to a cloud - based computer for remote data processing and operation. The data associated with the devices 1a - 1n can also be transferred to the local computer system 20063 for local data processing and operation. The modular devices 2a - 2m arranged in the same operating room may also be connected to the network switch 20062. The network switch 20062 can be connected to the network hub 20061 and / or the network router 20066 to connect the devices 2a - 2m to the cloud 20064. The data associated with the devices 2a - 2m can be transferred via the network router 20066 to the cloud computing system 20064 for data processing and operation. The data associated with the devices 2a - 2m may also be transferred to the local computer system 20063 for local data processing and operation.

[0087] The wearable sensing system 20011 can include one or more sensing systems 20069. The sensing system 20069 can include an HCP sensing system and / or a patient sensing system. One or more sensing systems 20069 can communicate with the computer system 20063 of the surgical hub system 20060 or the cloud server 20067 directly via one of the network routers 20066 or via the network hub 20061 or network switching 20062 that communicates with the network router 20066.

[0088] The sensing system 20069 can be coupled to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 can be transferred via the network router 20066 to the cloud computing system 20064 for data processing and manipulation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0089] As illustrated in FIG. 4, the surgical hub system 20060 can be extended by interconnecting a plurality of network hubs 20061 and / or a plurality of network switches 20062 with a plurality of network routers 20066. The modular communication hub 20065 can be housed within a modular control tower configured to receive a plurality of devices 1a - 1n / 2a - 2m. The local computer system 20063 may also be housed within the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images obtained by some of the devices 1a - 1n / 2a - 2m, for example, during a surgical procedure. In various aspects, the devices 1a - 1n / 2a - 2m can include various modules such as an imaging module coupled to an endoscope, a generator module coupled to an energy-based surgical device, a smoke evacuation module, a suction / irrigation module, a communication module, a processor module, a storage array, a surgical device coupled to a display, and / or a non-contact sensor module, among other modular devices that can be coupled to the modular communication hub 20065 of a surgical data network.

[0090] In one aspect, the surgical hub system 20060 illustrated in FIG. 4 may comprise a combination of a network hub(s), network switch, and network router(s) that connect devices 1a-1n / 2a-2m, or sensing system 20069, to the cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or sensing system 20069 connected to network hub 20061 or network switch 20062 may collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term "cloud" may be used as a metaphor for the "Internet," but this term is not so limited. Thus, the term "cloud computing" can be used herein to refer to "one type of Internet-based computing," in which case various services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 20065 and / or computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located within one or more operating rooms. Cloud computing services may perform a number of calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located within the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0091] By applying cloud computing data processing technology to the data collected by devices 1a to 1n / 2a to 2m, the surgical data network can provide improvements in surgical outcomes, cost reduction, and patient satisfaction. After tissue sealing and cutting procedures, at least some of devices 1a to 1n / 2a to 2m can be used to observe the tissue state to evaluate leakage or perfusion of the sealed tissue. At least some of devices 1a to 1n / 2a to 2m can be used to examine data including images of samples of body tissues for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of diseases. This can include tissue localization, margin confirmation, and phenotype. At least some of devices 1a to 1n / 2a to 2m can be used to identify the anatomical structures of the body using various sensors integrated with the imaging device and techniques such as overlaying images captured by multiple imaging devices. The data collected by devices 1a to 1n / 2a to 2m, including image data, can be transferred to cloud computing system 20064 or local computer system 20063 or both for data processing and operations including image processing and manipulation. The data may be analyzed to improve the results of surgery by determining whether further treatments such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and precision robotics can be performed on tissue-specific sites and conditions. Such data analysis may further employ prognostic analysis processing, and using standardized methods can provide useful feedback either to confirm surgical treatment and surgeon behavior or to propose modifications to surgical treatment and surgeon behavior.

[0092] Applying cloud computer data processing technology to the measurement data collected by the sensing system 20069 can result in improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing system 20069 may be used to evaluate the physiological state of a surgeon operating on a patient, a patient being prepared for surgery, or a patient recovering after surgery. The cloud-based computing system 20064 can monitor biomarkers associated with the surgeon or patient in real time, generate a surgical plan based at least on the measurement data collected before surgery, supply control signals to surgical instruments during surgery, and be used to notify the patient of complications during the postoperative period.

[0093] The operating room devices 1a to 1n can be connected to the modular communication hub 20065 via a wired channel or a wireless channel, depending on the configuration of the devices 1a to 1n with respect to the network hub 20061. In one aspect, the network hub 20061 may be implemented as a local network broadcast device that functions on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a to 1n located within the same operating room network. The network hub 20061 can collect data in the form of packets and transmit them to the router in half-duplex mode. The network hub 20061 cannot store any media access control / Internet Protocol (MAC / IP) for transferring device data. Only one of the devices 1a to 1n can transmit data at a time via the network hub 20061. The network hub 20061 cannot have a routing table or intelligence regarding the destination of information and broadcasts all network data across each connection and to the remote server 20067 of the cloud computing system 20064. The network hub 20061 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose a security risk and cause bottlenecks.

[0094] The operating room devices 2a to 2m can be connected to the network switch 20062 via a wired channel or a wireless channel. The network switch 20062 functions within the data link layer of the OSI model. The network switch 20062 may be a multicast device for connecting the devices 2a to 2m arranged in the same operating room to the network. The network switch 20062 transmits data in the form of frames to the network router 20066 and can function in full-duplex mode. Multiple devices 2a to 2m can transmit data simultaneously via the network switch 20062. The network switch 20062 stores and uses the MAC addresses of the devices 2a to 2m for transferring data.

[0095] The network hub 20061 and / or the network switch 20062 can be connected to the network router 20066 to connect to the cloud computing system 20064. The network router 20066 functions within the network layer of the OSI model. The network router 20066 creates a route for transmitting the data packets received from the network hub 20061 and / or the network switch 20062 to the cloud-based computer resources for further processing and operation of the data collected by any one or all of the devices 1a to 1n / 2a to 2m and the wearable sensing system 20011. The network router 20066 may be used to connect two or more different networks located at different positions, such as different operating rooms in the same medical facility or different networks in different operating rooms of different medical facilities. The network router 20066 transmits data in the form of packets to the cloud computing system 20064 and can function in full-duplex mode. Multiple devices can transmit data simultaneously. The network router 20066 can use IP addresses for transferring data.

[0096] In one example, the network hub 20061 may be implemented as a USB hub that enables a plurality of USB devices to be connected to a host computer. The USB hub can expand a single USB port into several levels so that there are more available ports for connecting devices to the host system computer. The network hub 20061 may include a wired function or a wireless function for receiving information via a wired channel or a wireless channel. In one aspect, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a to 1n and devices 2a to 2m located in the operating room.

[0097] In an example, the operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 may communicate with the modular communication hub 20065 via the Bluetooth wireless technology standard to exchange data over a short distance from fixed and mobile devices (using short - wavelength UHF radio waves in the 2.4 - 2.485 GHz ISM band) and to construct a personal area network (PAN). The operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 may communicate with the modular communication hub 20065 via some wireless communication standards or wired communication standards or protocols, such as Bluetooth, Low - Energy Bluetooth, near - field communication (NFC), Wi - Fi (IEEE802.11 family), WiMAX (IEEE802.16 family), IEEE802.20, New Radio (NR), Long - Term Evolution (LTE), and also Ev - DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, as well as any other wireless protocols and wired protocols designated for 3G, 4G, 5G, and beyond, but not limited to these. The computing module may include a plurality of communication modules. For example, the first communication module may be dedicated to short - range wireless communication such as Wi - Fi and Bluetooth, Low - Energy Bluetooth, Bluetooth Smart, etc., and the second communication module may be dedicated to long - range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev - DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, etc.

[0098] The modular communication hub 20065 functions as a central connection for one or more of the operating room devices 1a - 1n / 2a - 2m and / or the sensing system 20069 and can handle data types known as frames. The frames can carry data generated by the devices 1a - 1n / 2a - 2m and / or the sensing system 20069. When a frame is received by the modular communication hub 20065, the frame is amplified and sent to the network router 20066, which can transfer this data to the cloud computing system 20064 or the local computer system 20063 by using a number of wireless communication standards or wired communication standards or protocols as described herein.

[0099] The modular communication hub 20065 may be used as a stand - alone device or may be connected to compatible network hubs 20061 and network switches 20062 to form a larger network. The modular communication hub 20065 can be a good option for networking the operating room devices 1a - 1n / 2a - 2m as it is generally easy to install, configure, and maintain.

[0100] FIG. 5 illustrates a logic diagram of a control system 20220 for a surgical instrument or surgical tool according to one or more aspects of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may be at hand, such as an imaging device, a surgical stapler, an energy device, an end cutter device, and may include surgical supplies specific to the procedure. For example, the surgical instrument may include any one of an electric stapler, an electric stapler generator, an energy device, a high energy device, a high energy Joe device, an end cutter clamp, an energy device generator, an in - operating - room imaging system, a smoke evacuation device, a suction irrigation device, a pneumoperitoneum system, etc. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221 including a processor 20222 and a memory 20223. For example, one or more of sensors 20225, 20226, 20227 provide real - time feedback to the processor 20222. A motor 20230 driven by a motor driver 20229 is operably coupled to a longitudinally movable displacement member to drive an I - beam knife element. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. The position information may be provided to a processor 20222 that may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the position of the firing member, the firing bar, and the I - beam knife element. An additional motor may be provided to the tool driver interface to control the firing of the I - beam, the movement of the closure tube, the rotation of the shaft, and the articulation movement. A display 20224 may display various operating states of the instrument and may include a touch - screen function for data input. The information displayed on the display 20224 may be overlaid with an image acquired via an endoscope imaging module.

[0101] The microcontroller 20221 may be any single-core or multi-core processor, such as those known by the product names of ARM Cortex from Texas Instruments. In one aspect, the main microcontroller 20221 may be, for example, a 256KB single-cycle flash memory or other non-volatile memory on-chip memory with a maximum of 40MHz, the details of which are available in the product datasheet, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle SRAM, an internal ROM with StellarisWare (registered trademark) software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels, and may be an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments.

[0102] The microcontroller 20221 may also include a safety controller with two controller-based families such as TMS570 and RM4x known by the product names of Hercules ARM Cortex R4 from Texas Instruments. The safety controller may be configured specifically for safety-critical applications of IEC61508 and ISO26262, among others, to provide a scalable performance, connectivity, and memory options while providing a high-level integrated safety mechanism.

[0103] The microcontroller 20221 may be programmed to perform various functions such as precise control over the speed and position of the knife and the articulation movement system. In one aspect, the microcontroller 20221 may include a processor 20222 and a memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the articulation movement section or the knife system. In one aspect, the motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be readily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, titled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT", published on October 19, 2017, which is hereby incorporated by reference in its entirety.

[0104] The microcontroller 20221 may be programmed to provide accurate control over the speed and position of the displacement member and the articulation movement system. The microcontroller 20221 may be configured to calculate a response within the software of the microcontroller 20221. The calculated response may be compared with the measured response of the actual system to obtain an "observed" response, which is used for actual feedback decision-making. The observed response may be a suitably adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0105] The motor 20230 may be controlled by a motor driver 20229 and can also be used by a surgical instrument or tool firing system. In various forms, the motor 20230 may be a brushed DC drive motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver comprising field effect transistors (FETs). The motor 20230 may be powered by a power supply assembly releasably attached to a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may include a battery that may include a number of battery cells connected in series that may be used as a power source to power the surgical instrument or tool. In certain situations, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cell may be a lithium-ion battery that may be connectable to and separable from the power supply assembly.

[0106] The motor driver 20229 may be the A3941 available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include a proprietary charge pump regulator, which supplies a full (>10V) gate drive to a battery voltage up to 7V, enabling the A3941 to operate with a reduced gate drive up to 5.5V. A bootstrap capacitor may be used to supply the battery supply voltage required for the N-channel MOSFET. The internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FETs can be protected from shoot-through by a resistor-adjustable dead time. The integrated diagnostics indicate low voltage, overtemperature, and power bridge abnormalities and can be configured to protect the power MOSFETs under most short-circuit conditions. Other motor drivers may be easily substituted for use in the tracking system 20228 with an absolute positioning system.

[0107] The tracking system 20228 can comprise a controlled motor drive circuit configuration with a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for an absolute positioning system can supply a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member can represent a longitudinally movable drive member having a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reduction assembly. In some examples, the displacement member can represent a firing member adapted and configured to include a rack of drive teeth. In some examples, the displacement member can represent a firing bar or an I-beam, each of which can be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member can generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, a firing member, a firing bar, an I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member can be coupled to the firing member, the firing bar, and the I-beam. Thus, the absolute positioning system can actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member can be coupled to any position sensor 20225 suitable for measuring linear displacement. Thus, the longitudinally movable drive member, the firing member, the firing bar, or the I-beam, or combinations thereof, can be coupled to any suitable linear displacement sensor. The linear displacement sensor can include a contact displacement sensor or a non-contact displacement sensor.The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system including a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system including a fixed magnet and a series of linearly arranged movable Hall effect sensors, an optical detection system including a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system including a fixed light source and a series of linearly arranged movable photodiodes or photodetectors, or any combination thereof.

[0108] The electric motor 20230 may include a rotatable shaft that operably interfaces with a gear assembly attached to engage a set of drive teeth or a rack on a displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 20225 element corresponds to some linear longitudinal translation of the displacement member. The configuration of the gear ring and sensor may be connected to a linear actuator by a rack and pinion configuration or to a rotary actuator by spur gears or other connections. A power supply may supply power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for engaging a corresponding drive gear of a gear reduction assembly. The displacement member may represent a longitudinally movable firing member, firing bar, I-beam, or a combination thereof.

[0109] One rotation of the sensor element associated with the position sensor 20225 can correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance that the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor device can be connected via a gear reduction device in which the position sensor 20225 completes one or more rotations with respect to the full stroke of the displacement member. The position sensor 20225 may complete multiple rotations with respect to the full stroke of the displacement member.

[0110] To provide a unique position signal for two or more rotations of the position sensor 20225, a series of switches (where n is an integer greater than 1) may be used alone or in combination with a gear reduction device. The state of the switch can be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1 + d2 +... dn of the displacement member. The output of the position sensor 20225 is supplied to the microcontroller 20221. The position sensor 20225 of the sensor device may comprise a magnetic sensor, an analog rotational sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0111] The position sensor 20225 may comprise any number of magnetic sensing elements such as, for example, a magnetic sensor classified by measuring the total magnetic field or a vector component of the magnetic field. The technologies used to produce both types of magnetic sensors can encompass numerous aspects of physics and electronics. Technologies used for sensing magnetic fields include, among others, search coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetic impedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and microelectromechanical system-based magnetic sensors.

[0112] The position sensor 20225 of the tracking system 20228 with an absolute positioning system may comprise a magnetic rotary absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 20225 is connected to the microcontroller 20221 to realize an absolute positioning system. The position sensor 20225 is a low-voltage and low-power component, and may include four Hall effect elements in the area of the position sensor 20225 that can be arranged above the magnet. Also, a high-resolution ADC and a smart power management controller may be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the border algorithm, may be provided to implement a concise and efficient algorithm for calculating hyperbolic and trigonometric functions that only requires addition, subtraction, bit shift, and table reference operations. The angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 20221 via a standard serial communication interface such as a serial peripheral interface (SPI) interface. The position sensor 20225 may provide a resolution of 12 bits or 14 bits. The position sensor 20225 may be an AS5055 chip provided in a small QFN16-pin 4×4×0.85 mm package.

[0113] Tracking system 20228 with an absolute positioning system may include and / or be programmed to implement a feedback controller such as a PID, state feedback, and adaptive controller. The power supply converts a signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by position sensor 20225, other sensor(s) may be provided to measure physical parameters of the physical system. In some aspects, examples of other sensor(s) may include sensor arrangements such as those described in U.S. Patent No. 9,345,481, issued May 24, 2016, titled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is hereby incorporated by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, titled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is hereby incorporated by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, titled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is hereby incorporated by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include a comparison and combination circuit to combine the calculated response with the measured response using algorithms such as weighted averaging and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, inductive resistance, etc. to predict how the state and output of the physical system will be based on knowing the input.

[0114] An absolute positioning system can provide the absolute position of a displacement member upon power-up of the instrument without having to retract or advance the displacement member to a reset (zero or home) position, which may require a conventional rotary encoder that simply counts the number of steps forward or backward taken by the motor 20230 to estimate the position of a device actuator, drive bar, knife, etc.

[0115] For example, a sensor 20226, such as a strain gauge or a micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as, for example, the closing force applied to the anvil, or the amplitude of the strain exerted on the anvil during the clamping operation. The measured strain may be converted into a digital signal and provided to the processor 20222. Instead of, or in addition to, the sensor 20226, a sensor 20227, such as a load cell, for example, may measure the closing force applied to the anvil by the closing drive system. For example, a sensor 20227, such as a load cell, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage a wedge thread, which is configured to cam the staple driver upward to eject staples and deformably contact the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue when the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 20231 may be used to measure the current consumed by the motor 20230. The force required to advance the firing member can correspond to, for example, the current drawn by the motor 20230. The measured force may be converted into a digital signal and provided to the processor 20222.

[0116] For example, a strain gauge sensor 20226 may be used to measure the force applied to tissue by an end effector. To measure the force exerted by the end effector on the tissue being treated, the strain gauge may be coupled to the end effector. A system for measuring the force applied to tissue grasped by the end effector may comprise a strain gauge sensor 20226, such as a micro strain gauge, configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 20226 can measure the amplitude or magnitude of the strain exerted on the jaw members of the end effector during a clamping operation, which can indicate tissue compression. The measured strain can be converted into a digital signal and supplied to a processor 20222 of a microcontroller 20221. A load sensor 20227 may measure, for example, the force used to operate a knife element to cut tissue captured between an anvil and a staple cartridge. A magnetic field sensor may be used to measure the thickness of the captured tissue. The measurements of the magnetic field sensor may also be converted into a digital signal and provided to the processor 20222.

[0117] The measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, respectively measured by sensors 20226, 20227, may be used by the microcontroller 20221 to characterize corresponding values of a selected position of the firing member and / or the velocity of the firing member. In one embodiment, the memory 20223 may store techniques, equations, and / or look-up tables that may be used by the microcontroller 20221 during evaluation.

[0118] The control system 20220 of the surgical instrument or tool may also comprise a wired or wireless communication circuit for communicating with a surgical hub 20065 as shown in FIG. 4.

[0119] FIG. 6 illustrates an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 that can communicate with a console 20294 or a portable device 20296 through a local area network 20292 and / or a cloud network 20293 via a wired and / or wireless connection. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 such that the adapter 20285 transmits force from the drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein for measuring the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in vivo loading unit, i.e., a multi-firing loading unit (MFLU), that allows a clinician to fire a plurality of fasteners multiple times without having to remove the loading unit 20287 from the surgical site to reload the loading unit 20287.

[0120] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include an exchangeable multi-firing fastener cartridge that includes a plurality of fasteners (e.g., staples, clips, etc.) that can be fired two or more times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener as the fastener is ejected from the multi-firing fastener cartridge.

[0121] The handle 20297 may include a motor coupled to the drive shaft so as to act on the rotation of the drive shaft. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanism or user interface, which may be engaged by a clinician to activate the motor.

[0122] The control interface of the handle 20297 may communicate with a controller 20298 of the handle 20297 to selectively activate the motor and act on the rotation of the drive shaft. The controller 20298 may be disposed within the handle 20297 and may be configured to receive inputs from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 may analyze the inputs from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display visible to the clinician during use of the handle 20297. The display may be configured to display portions of the adapter or loading unit data before, during, or after firing of the instrument 20282.

[0123] The adapter 20285 may include an adapter identification device 20284 disposed therein, and the loading unit 20287 may include a loading unit identification device 20288 disposed therein. The adapter identification device 20284 may communicate with the controller 20298, and the loading unit identification device 20288 may communicate with the controller 20298. It will be understood that the loading unit identification device 20288 may communicate with the adapter identification device 20284 that relays or passes the communication from the loading unit identification device 20288 to the controller 20298.

[0124] Adapter 20285 may also include a plurality of sensors 20286 (one is shown) disposed therearound to detect various states of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of firings of the adapter 20285, the peak force of the adapter 20285 during firing, the total amount of force applied to the adapter 20285, the peak recoil force of the adapter 20285, the number of rest periods of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide an input to the adapter identification device 20284 in the form of a data signal. The data signals of the plurality of sensors 20286 may be stored within the adapter identification device 20284 or may be used to update the adapter data stored within the adapter identification device 20284. The data signals of the plurality of sensors 20286 may be analog or digital. The plurality of sensors 20286 may include a force gauge for measuring the force exerted on the loading unit 20287 during firing.

[0125] The handle 20297 and the adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally, or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting (e.g., inductively transmitting) energy and signals therebetween. It is also contemplated that the adapter identification device 20284 and the controller 20298 may wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0126] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., the LAN 20292, the cloud 20293, the console 20294, or the portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 20280. For example, the controller 20298 may transmit instrument data including the serial number of a mounting adapter (e.g., adapter 20285) attached to the handle 20297, the serial number of a loading unit (e.g., loading unit 20287) attached to the adapter 20285, and the serial numbers of a plurality of firing fastener cartridges loaded into the loading unit to the console 20294. The console 20294 may then reply to the controller 20298 with data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively. The controller 20298 may display a message on the local instrument display, or transmit a message to the console 20294 or the portable device 20296 via the transceiver 20283 to display the message on the display 20295 or the portable device screen, respectively.

[0127] FIG. 7 illustrates a diagram of a situation awareness surgical system 5100 according to at least one aspect of the present disclosure. The data source 5126 can include, for example, a modular device 5102 (which can include sensors configured to detect parameters associated with a patient, an HCP, and the environment, and / or the modular device itself), a database 5122 (e.g., an EMR database including patient records), a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor), an HCP monitoring device 35510, and / or an environment monitoring device 35512. The surgical hub 5104 can be configured to derive context information regarding a surgical procedure from the data, for example, based on a particular combination of the received data or the particular order in which data is received from the data source 5126. The context information inferred from the received data can include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure that the surgeon is performing, the type of tissue being operated on, or the body cavity that is the subject of the procedure. This function according to some aspects of the surgical hub 5104 for deriving or inferring information regarding a surgical procedure from the received data can be referred to as "situation awareness." For example, the surgical hub 5104 can incorporate a situation awareness system, which is the hardware and / or programming associated with the surgical hub 5104 that derives context information regarding a surgical procedure from received data and / or surgical planning information received from an edge computing system 35514 or a corporate cloud server 35516.

[0128] The situation recognition system of the surgical hub 5104 can be configured to derive context information from data received from various different data sources 5126. For example, the situation recognition system can include a pattern recognition system, or a machine learning system (such as an artificial neural network), trained with training data, to correlate various inputs (such as data from the database 5122, the patient monitoring device 5124, the modular device 5102, the HCP monitoring device 35510, and / or the environmental monitoring device 35512) with corresponding context information regarding the surgical procedure. The machine learning system can be trained to accurately derive context information regarding the surgical procedure from the provided inputs. In an example, the situation recognition system can include a lookup table that stores pre-characterized context information regarding the surgical procedure in association with one or more inputs (or a range of inputs) corresponding to that context information. In response to a query with one or more inputs, the lookup table can return the corresponding context information of the situation recognition system to control the modular device 5102. In an example, the context information received by the situation recognition system of the surgical hub 5104 can be associated with specific control adjustments, or a series of control adjustments, of one or more modular devices 5102. In an example, the situation recognition system can include a further machine learning system, a lookup table, or other such system that generates or reads one or more control adjustments of one or more modular devices 5102 when context information is provided as an input.

[0129] The surgical hub 5104 incorporating the situation awareness system can provide many advantages to the surgical system 5100. One advantage can include providing improved interpretation of sensed and collected data, which can improve the processing accuracy during the surgical procedure and / or the use of the data. Returning to the previous example, the situation awareness surgical hub 5104 can determine which type of tissue is being operated on, and thus, if an unexpectedly high force is detected to close the end effector of the surgical instrument, the situation awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the tissue type.

[0130] The type of tissue being operated on can affect the adjustments made to the compression speed and load threshold of a surgical stapling and cutting instrument for specific tissue gap measurements. The situation awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic procedure or an abdominal procedure, whereby the surgical hub 5104 can determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (in the case of a thoracic procedure) or stomach tissue (in the case of an abdominal procedure). The surgical hub 5104 can then appropriately adjust the compression speed and load threshold of the surgical stapling and cutting instrument according to the type of tissue.

[0131] The type of body cavity being operated on during a insufflation procedure can affect the function of the smoke evacuation device. The situation awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure utilizes insufflation) and can determine the type of procedure. Generally, since a certain type of procedure can be performed within a specific body cavity, the surgical hub 5104 can appropriately control the motor speed of the smoke evacuation device according to the body cavity being operated on. Thus, the situation awareness surgical hub 5104 can provide a consistent amount of smoke evacuation for both thoracic and abdominal procedures.

[0132] The type of procedure being performed can affect the energy level optimal for the operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopic procedures, the end effector of an ultrasonic surgical instrument or an RF electrosurgical instrument is immersed in fluid, which may require a higher energy level. The Situational Awareness Surgical Hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The Surgical Hub 5104 can then adjust the RF power level of the generator or the ultrasonic amplitude (e.g., "energy level") to compensate for the fluid-filled environment. In connection therewith, the type of tissue being operated on can affect the energy level optimal for the operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The Situational Awareness Surgical Hub 5104 can determine which type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument, respectively, according to the tissue shape expected for the surgical procedure. Further, the Situational Awareness Surgical Hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument not only for each procedure but also over the course of the surgical procedure. The Situational Awareness Surgical Hub 5104 can determine which step of the surgical procedure is being performed or will be performed next and then update the control algorithm of the generator and / or the ultrasonic surgical instrument or the RF electrosurgical instrument to set the energy level to an appropriate value for the type of tissue expected according to the steps of the surgical procedure.

[0133] In an example, the surgical hub 5104 can derive data from an additional data source 5126 to improve the conclusions drawn from one data source 5126. The situation awareness surgical hub 5104 can enhance the data received from the modular device 5102 with context information constructed regarding the surgical procedure from other data sources 5126. For example, the situation awareness surgical hub 5104 can be configured to determine whether hemostasis has occurred (e.g., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. The surgical hub 5104 can be further configured to compare a physiological measurement (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device communicatively coupled to the surgical hub 5104) to make a determination regarding the integrity of the staple line or tissue adhesion. The situation awareness system of the surgical hub 5104 can provide additional context when analyzing the visualization data, taking into account the physiological measurement data. The additional context can be useful when the visualization data may not be conclusive or may be incomplete by itself.

[0134] For example, if it is determined that the use of an instrument is required in a subsequent step of the procedure, the situation awareness surgical hub 5104 can actively activate the generator to which the RF electrosurgical instrument is connected. Actively activating the energy source can make it possible to have the instrument ready for use as soon as the preceding step of the procedure is completed.

[0135] The situation awareness surgical hub 5104 can determine whether the current step or subsequent steps of a surgical procedure require different views or magnifications on a display according to the features of the surgical site that a surgeon is expected to view. The surgical hub 5104 can actively change the displayed view (e.g., supplied from a medical imaging device for a visualization system) as appropriate, whereby the display automatically adjusts throughout the surgical procedure.

[0136] The situation awareness surgical hub 5104 can determine which step of a surgical procedure is being performed or will be performed next and whether specific data or a comparison between data is required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up a data screen based on the step of the surgical procedure being performed without waiting for a surgeon to request specific information.

[0137] During the setup of a surgical procedure or during the surgical procedure itself, errors can be checked. For example, the Situational Awareness Surgical Hub 5104 can determine whether the operating room is appropriately or optimally set up for the surgical procedure to be performed. The Surgical Hub 5104 can determine the type of surgical procedure being performed, read the corresponding checklist, product locations, or setup requirements (e.g., from memory), and then be configured to compare the current operating room layout to the standard layout for the type of surgical procedure that the Surgical Hub 5104 has determined is being performed. In some examples, the Surgical Hub 5104 can compare a list of items for the procedure and / or a list of devices paired with the Surgical Hub 5104 to the recommended or expected manifest of items and / or devices for a given surgical procedure. If there is any discrepancy between the lists, the Surgical Hub 5104 can provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, HCP monitoring device 35510, environmental monitoring device 35512, and / or other surgical supplies are missing. In some examples, the Surgical Hub 5104 can determine, for example, the relative distance or relative position of the modular device 5102 and the patient monitoring device 5124 via a proximity sensor. The Surgical Hub 5104 can compare the relative position of the devices to the layout recommended or expected for a particular surgical procedure. If there is any discrepancy between the layouts, the Surgical Hub 5104 can be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0138] The situation awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making a mistake or deviating from a series of actions expected during a surgical procedure. For example, the surgical hub 5104 can determine the type of surgical procedure being performed, read a corresponding list of steps or order of device use (e.g., from memory), and then compare the steps being taken or devices being used during the surgical procedure to the steps or devices expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. The surgical hub 5104 can provide an alert indicating that an unexpected action is being taken at a particular step in the surgical procedure or that an unexpected device is being utilized.

[0139] Surgical instruments (and other modular devices 5102) can be adjusted to suit the specific context of each surgical procedure (such as adjustment to different tissue types) and can verify actions during the surgical procedure. The next steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the procedure.

[0140] Surgical autonomous systems, devices, and methods can include aspects of integration with other medical devices, data sources, processes, and institutions. Surgical autonomous systems, devices, and methods can include, for example, aspects of integration with a computer-implemented bidirectional surgical system and / or one or more elements of a computer-implemented bidirectional surgical system. Surgical systems, surgical autonomous systems, and autonomous surgical systems can be compatible as described herein.

[0141] Referring to FIG. 8, an overview of a surgical autonomous system 48000 can be provided. Surgical instrument A 48010 and / or surgical instrument B 48020 can be used in a surgical operation as part of the surgical system 48000. The surgical hub 48035 can also be configured to regulate the information flow to the display of the surgical instrument. For example, the surgical hub may be described in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. Exemplary surgical instruments suitable for use with the surgical system 48000 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385), filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0142] FIG. 8 shows an example of a surgical autonomous system 48000. The system 48000 can be used to perform a surgical operation on a patient lying on an operating table in an operating room. A robotic system can be used as part of the surgical system in a surgical operation. For example, the robotic system may be described in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety. A robotic hub can be used to process an image of the surgical site and then display it to the surgeon through the surgeon's console.

[0143] Other types of robotic systems can be readily adapted to be used with the surgical system 48000. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019 / 0201137 (A1) (U.S. Patent Application No. 16 / 209,407), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0144] Various examples of cloud-based analysis methods implemented by the cloud and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569 (A1) (U.S. Patent Application No. 16 / 209,403), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB", filed on December 4, 2018, the disclosure of which is hereby incorporated by reference in its entirety.

[0145] In various aspects, the imaging device may be used in a surgical system and may include at least one image sensor and one or more optical components. Suitable image sensors can include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0146] The optical components of the imaging device may include one or more light sources and / or one or more lenses. The one or more light sources can be directed to illuminate a portion of the surgical field. The one or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0147] One or more light sources may be configured to irradiate electromagnetic energy within the visible spectrum as well as the invisible spectrum. The visible spectrum is sometimes also referred to as the optical spectrum or emission spectrum and is a portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by the human eye) and may be referred to as visible light, or simply light. A typical human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0148] The invisible spectrum (e.g., non-emission spectrum) is a portion of the electromagnetic spectrum that is located below and above the visible spectrum (i.e., wavelengths less than approximately 380 nm and greater than approximately 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than approximately 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than approximately 380 nm are shorter than the violet spectrum and become invisible ultraviolet, x-rays, and gamma ray electromagnetic radiation.

[0149] In various aspects, the imaging device may be configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-ureteroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0150] The imaging device may employ multispectral monitoring to distinguish topography from the underlying structure. A multispectral image captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments with sensitivity to specific wavelengths, including frequencies beyond the visible light range, such as IR and light from ultraviolet. Spectral imaging enables extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed Dec. 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after a surgical task for performing one or more of the above-described tests on the treated tissue has been completed. It is understood that strict sterilization of the operating room and surgical instruments is required during any surgical procedure. The strict hygiene and sterilization conditions required in the "operating room," i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize anything that comes into contact with the patient or enters the sterile field, including the imaging device and its accessories and components. It will be understood that the sterile field can be considered a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding a patient prepared for surgery. The sterile field can include properly attired and scrubbed team members, as well as all equipment and fixtures within that area.

[0151] Surgical instrument A 48010 and / or surgical instrument B 48020 may have one or more capabilities. (For example, surgical instrument A has capabilities B, Z, and D, and surgical instrument B has capabilities C, F, and E.) The capabilities may be associated with functions that the surgical instruments can perform (e.g., autonomous function A 48015 associated with surgical instrument A and autonomous function B 48025 associated with surgical instrument B). Examples of functions may be described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019 / 0200844 (A1) (U.S. Patent Application No. 16 / 209,385) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. For example, if the surgical instrument is an end cutter, one of the capabilities may be to excise tissue (e.g., the tissue surrounding the colon when a surgeon performs a colectomy). The capabilities may include surgical tasks as described with respect to FIG. 9. For example, the ability to excise tissue may include control of an energy source, cutting, stapling, knob orientation, body orientation, body position, anvil jaw force, reload alignment slot management, and / or the like.

[0152] Surgical instrument A and / or B may be associated with control feedback or fault mitigation feedback as described herein. For example, a level of autonomy with minimal manual input may be associated with control feedback. For example, a level of autonomy with increased manual input may be associated with fault mitigation feedback.

[0153] Data can be generated based on the performance of surgical instrument A and / or B (e.g., by a monitoring module located at surgical hub 48035 or locally by a surgical instrument as described with respect to FIG. 11). The data can be related to how the current autonomy level at which the surgical instrument is operating is performing with respect to performance. For example, the data may be associated with body measurements, physiological measurements, and / or equivalents as described with respect to FIGS. 10 and 11. The measurements are described in more detail under the title "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements" in U.S. Patent Application No. 17 / 156,28, filed Nov. 10, 2021, the disclosure of which is hereby incorporated by reference in its entirety.

[0154] The indication 48045 of the data may be transmitted, for example, to a surgical hub 48035 where it can be evaluated. In an example, the indication 48045 may be transmitted to a cloud service (e.g., Amazon Web Services) as described herein. The data can be used as an input in an analysis module, for example, to check whether the performance of the surgical instrument is within an acceptable range (e.g., exceeding a failure threshold 48040). The indication can be associated with switching from using control feedback 48005 to failure mitigation feedback.

[0155] FIG. 9 shows the relationship between control feedback 48060 and fault mitigation feedback 48095. As shown in FIG. 9, the device can include an expertise model 48055. The expertise model 48055 can provide parameter(s) so that the surgical instrument can operate autonomously. For example, the surgical instrument can autonomously perform one or more surgical tasks (e.g., it may be possible to perform). For example, the expertise model 48055 may be linked to a machine learning algorithm. The machine learning algorithm may include a neural network structure that can generate parameter(s) (e.g., weights of the parameters) used on a script disposed inside the surgical instrument, as described herein. In the example, the script may be remotely executed, for example, at a surgical hub. The parameters output by the expertise model 48055 and executed by the script may be dynamic. The script may be used to execute one or more actuators disposed on the surgical instrument, as described with respect to FIG. 6. For example, the one or more actuators may be linear actuators and / or rotary actuators.

[0156] The actuator may be a servo motor disposed on the surgical instrument. The script may be generated using a machine learning algorithm with adjustable parameters and may be used in association with hardware such as linear and rotary actuators to enable the surgical instrument to perform a surgical task. For example, the script may generate control feedback 48060 that can be used as an input to the actuator. The actuator may enable the surgical instrument to be autonomously controlled via the script. For example, the actuator can receive adjustable parameters as inputs and use those parameters to autonomously perform a surgical task.

[0157] One or more physical reality measurement values 48085 can be obtained. For example, the physical reality measurement value 48085 can be associated with one or more sensors disposed on a surgical instrument that performs a surgical task. In an example, the physical reality measurement value 48085 can be obtained and / or generated from a patient wearing a wearable sensor that measures a biomarker of the patient. In an example, the physical reality measurement value 48085 can be generated from a surgeon wearing a wearable sensor that measures a biomarker and other health markers of the surgeon.

[0158] The physical reality measurement value 48085 may be transmitted to a surgical hub and may be analyzed at the surgical hub. In an example, the physical reality measurement value 48085 can be locally generated at a surgical instrument, for example, by a surgical instrument processor, as described herein. In an example, the surgical instrument and / or the surgical hub can include an analysis module that can analyze the physical reality measurement value 48085. When the physical reality measurement value 48085 is generated by a remote source such as a surgical hub, the physical reality measurement value 48085 may be transmitted to the surgical instrument via a message. In an example, the message can pass through a surgical application programming interface (API). The physical reality measurement value 48085, or in other words, the data associated with the physical reality measurement value, may be compared to a model measurement value (e.g., model data), as described herein.

[0159] The physical reality measurement value 48085 can be converted into the measured reality data 48075. For example, the physical reality measurement value 48085 may pass through a surgical hub or an analysis module disposed locally on a surgical instrument. The output of the analysis module may be data related to the measured reality. For example, the physical reality measurement value 48085 may be raw data related to the position of a linear and / or rotational actuator of a surgical instrument. For example, this data may be in the form of a voltage reading. In an example, an analog-to-digital converter (ADC) may be included, which may convert the voltage reading into a bit stream, and the bit stream may be used as the physical reality measurement value 48085. This raw data may be transmitted to a surgical hub or to an analysis module disposed locally on a surgical instrument, and the analysis module may output the measured reality data 48075. The measured reality data 48075 may be in a form more suitable for comparison when compared to the raw physical reality measurement value 48085.

[0160] The measured reality data 48075 can be compared with data associated with an expertise model. For example, the expertise model can include a data structure that holds model data. The model data may sometimes be referred to herein as model reality data. This model reality data may be compared with the measured reality data 48075, and a difference 48070 between the two may be generated. This difference 48070 between the model reality data and the measured reality data 48075 is used as control feedback 48065 and may adjust parameters for controlling servo motors such as the linear and rotational actuators described herein.

[0161] For example, the difference 48070 between the model reality data and the measured reality data may indicate that the speed at which the rotary actuator rotates is too high. The measured reality data 48075 associated with the rotational speed may be greater than the model reality data associated with the rotational speed of the rotary actuator A positioned at a specific spot on the surgical instrument. Based on this indication, control feedback 48065 may be generated, and the parameter associated with the speed of the rotary actuator A may be decreased so that the speed of the rotary actuator decreases.

[0162] The expertise model 48055 can be used, for example, to generate control feedback 48065 for the control of the actuator of the autonomously operating surgical instrument as long as the difference 48070 between the modeled reality measurement data and the measured reality data 48075 is within a threshold. If the difference 48070 between these two does not fall within the threshold 48115 (e.g., exceeds the threshold 48115), a fault mitigation model 48090 can be used (e.g., a control loop 48110 grounded to 0 and a fault mitigation loop release 48105 grounded from 0).

[0163] The fault mitigation model 48090 can generate parameters used by the autonomously operating surgical instrument. The parameters can be set to values associated without risk. In the example, the fault mitigation model 48090 can change the surgical instrument from autonomous settings to manual settings. The surgical instrument using the fault mitigation parameters can control the linear and / or rotary actuator and other motor controls based on the fault mitigation feedback 48095.

[0164] The physical reality measurement value 48085 can be generated to evaluate the performance of the fault reduction control 48100. Similar to the expertise model 48055, the physical reality measurement value 48085 may be in the form of raw data or may pass through the analysis module. The analysis module can generate the measured reality data 48075, which can be used and compared with the fault model data. The measured reality data 48075 associated with the fault reduction parameters can be compared with the fault model reality data, for example, to evaluate the performance of the fault reduction model 48090.

[0165] Notification of the switch from the expertise model 48055 to the fault reduction model 48090 can be sent to the user of the surgical instrument via a message. The message may include the parameters set for the surgical instrument. In the example, when comparing the measured reality data 48075 with the model reality data, the patient's heart rate can be compared. This data related to the heart rate may be checked against the threshold 48115. If the heart rate exceeds the threshold, the surgical instrument may switch from using the model of the expertise script to the fault reduction model script. The fault reduction model 48090 can generate fault reduction parameters. The parameters may be known to result in the surgical instrument being able to perform the surgical task without risk.

[0166] FIG. 10 shows an example of executing an autonomous function using an ideal model 48120 and a failure model 48145. As shown in FIG. 10, the ideal model 48120 may include one or more surgical tasks. The ideal model 48120 and the expertise model may be used interchangeably herein. The ideal model 48120 may be generated by a surgical hub. For example, the ideal model 48120 may be generated by a surgical hub as described with respect to FIG. 9. The surgical tasks of the ideal model (e.g., each of the surgical tasks), such as surgical task 1 48130, surgical task 2, and surgical task K 48135, may be associated with one or more ideal metrics. The ideal metrics may be associated with ideal physiological and / or physical metrics related to the execution of the surgical tasks. For example, the surgical tasks may be executed autonomously. The ideal metrics may be related to the physiological and / or physical expected states that a surgeon may expect to exist based on the surgical tasks. For example, the surgical task may be to remove surrounding tissue from an organ such as the colon. The ideal physical state and / or metrics for this surgical task may include that the patient's heart rate exceeds a specific value and is less than a specific value (e.g., within an acceptable range). This may be regarded as the ideal heart rate range for mobilizing the colon. In the example, the surgical task may be to excise tissue, which may include the use of an end cutter. Excising tissue may include one or more instrument capabilities to be executed, such as positioning the instrument body, controlling the anvil jaw force, and managing the reload alignment slot. These capabilities may be executed autonomously. For example, the first autonomous function may be associated with the expertise model described herein (e.g., a software module disposed therein). To execute the capabilities for the surgical task of tissue excision, the expertise model may execute the first autonomous function (e.g., a script of the autonomous function). Based on the fact that a failure reduction threshold has been exceeded, the second autonomous function may be called. The second autonomous function may be associated with the failure reduction model (e.g., a software module that may be disposed within the failure reduction model).To perform the capabilities for the surgical task of tissue resection, the expertise model may switch from the execution of a first autonomous function (e.g., the script of the first autonomous function) to the execution of a second autonomous function (e.g., the script of the second autonomous function).

[0167] The physical and / or physiological ideal state can vary based on the surgical task. For example, surgical task 2 may have different ideal physiological and / or physical states when compared to surgical task 1. Physical metrics may include data generated from sensors from one or more actuators, as described with respect to FIGS. 9 and 12. For the surgical instrument to perform an autonomous function, the surgical instrument may perform surgical task 1 48130, surgical task 2, and / or surgical task K 48135. The ideal physical and / or physiological state may be compared to the measured physiological and / or physical state. For example, as shown with respect to the measured autonomous function 48160, the measured physiological and / or physical state may be generated from one or more of data generated from sensors, actuators, robotic functions, data generated from the patient such as physiological data, data generated from the surgeon such as physiological surgeon data (48155).

[0168] Data generated (48155) from the patient and / or surgeon may involve the use of wearable devices to measure biomarkers associated with the patient and / or surgeon. Wearables are described in more detail under the title "Monitoring of adjusting a surgical parameter based on biomarker measurements" in U.S. Patent Application No. 17 / 156,28, filed on November 10, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0169] This data can be used as input for generating and (48155) collecting physiological and / or physical metrics. Data associated with ideal physical and / or physiological metrics may be used and compared to data generated from the measured physical and / or physiological state (e.g., for each of surgical tasks such as surgical task 1 48165, surgical task 2, and / or surgical task K 48170 associated with the measured autonomic function 48160), a difference between these two may be generated and compared to a threshold 48180, which may be done at the surgical hub or locally at the surgical instrument as described with respect to FIGS. 9 and 12.

[0170] For example, determining a threshold 48180 against which the difference between an ideal physical and / or physiological state and a measured physical and / or physiological state is compared may be generated using a machine learning model 48185 that may include a neural network structure. The neural network structure may use one or more ideal physical and / or physiological states as parameters when determining an appropriate threshold level 48180. As described with respect to FIG. 12, for example, a threshold level 48180, which may be referred to as a fault reduction threshold level 48180, may cause the surgical instrument to switch to using a fault model 48145 if exceeded based on the difference between the ideal physiological and physical states and the measured physical and physiological states.

[0171] Fault model 48145 may involve the surgical instrument being associated with fault reduction feedback. Fault model 48145 can include physiological fault metrics and / or physical fault metrics such as the physical and / or physiological state of the fault. Similar to the ideal model, these conditions can be compared to the measured physiological and physical states described herein. Determining both the ideal physical and physiological states and the faulty physical and physiological states may involve the use of a machine learning model 48175 as described herein. For example, a machine learning model 48175 associated with a fault reduction model 47145 can determine a fault reduction metric that eliminates or minimizes any risk associated with performing one or more surgical tasks (e.g., surgical tasks 1, 2, and / or K).

[0172] The machine learning model 48125 associated with the ideal model can determine one or more physical and / or physiological metrics associated with optimized performance of the surgical task. The optimized performance may take into account the costs and / or benefits associated with the surgical task. For example, surgical task 1 may include the surgeon mobilizing the patient's colon. During the surgical task, the surgeon's heart rate may have an ideal physiological measurement. The patient's heart rate may have an ideal physiological measurement. The position of the instrument may have an ideal physical measurement that can be determined by the position of one or more actuators. In the example, this data may be analyzed and values may be generated. In the example, each of the ideal data collected from the physiological state and / or physical state may be compared to the respective measured physiological state and / or physical state. For example, the ideal heart rate of the surgeon may be compared to the measured heart rate of the surgeon as determined during surgical task 1 of the autonomic function. The patient's heart rate may be compared to the patient's heart rate measured during surgical task 1 48130. In the example, if one of the differences between the physical state of the ideal model and the measured physical state and / or physiological state of the model exceeds the fault reduction threshold 48180, the surgical instrument may switch from operating on the control feedback 48140 to operating on the fault reduction feedback 48141. The fault reduction feedback 48141 can be described with respect to FIG. 9. In the example, when switching from surgical task 1 48130 to surgical task 2, a set of ideal physical and / or physiological metrics can be generated by the ideal model that can be placed on the surgical instrument and / or on the surgical hub.

[0173] In an example, the surgical instrument can pull data from a remote source and generate an ideal metric associated with surgical task 2, which can be based on past data associated with surgical task 2. The past data may be based on the past data of the facility or on general past data associated with surgical task 2. The physical and / or physiological state, and / or the metric associated with the physical and / or physiological state, may be manually input by the surgeon or another user of the surgical instrument.

[0174] FIG. 11 shows an example of performing an autonomous function using control feedback and fault mitigation feedback. An overview 48195 of performing an autonomous function using control feedback and fault mitigation feedback can be provided. The autonomous function and the autonomous surgical task can be used interchangeably herein.

[0175] At 48200, a first autonomous function can be performed. The first autonomous function can be associated with an expertise model as described with respect to FIG. 9. For example, the model of expertise can output parameters that are used when the autonomous function is performed. The autonomous function may perform a surgical task associated with the surgical instrument. The control of the surgical instrument, such as the actuator described with respect to FIGS. 9 and 12, may be controlled by control feedback. The control feedback can update the parameters based on the current performance of the autonomous function.

[0176] The control feedback can be generated when comparing the modeled real-world data associated with the expertise model with the measured real-world data generated based on the physical and / or physiological measurements associated with the execution of the surgical task. For example, the physical measurement may include the position of the actuator(s) disposed on the surgical instrument performing the surgical task.

[0177] In 48205, a first autonomous function that can be associated with an expertise model can be monitored. For example, one or more outputs can be generated based on the execution of the autonomous function.

[0178] The output can be a physical measurement or in the form of raw data. The output can be supplied to an analysis module locally at a surgical hub or in a surgical instrument. The output can include data generated based on situation awareness. For example, surgical context data can be sent to the surgical hub. The data generated based on situation awareness can include any data suitable for characterizing the current presentation of the patient considering the ongoing surgical procedure. For example, such data can include treatment stage, current operating room (OR) configuration, patient vitals, instruments, personnel, and / or the like. The surgical hub can use surgical context data when generating measured real-world data. The surgical context data can be sent to the analysis module, for example, together with physical measurement data.

[0179] The expertise model can generate parameters by using a machine learning model. The parameters can be used to execute the autonomous function. When generating the parameters used for the autonomous function, the machine learning model can consider one or more of past data, surgical context, type of the first autonomous function, type of the second autonomous function, past data associated with the first autonomous function, number of outputs exceeding a failure threshold, or degree of one or more parameters. The threshold and the failure threshold can be used interchangeably herein. One or more of the following described herein can be applied when determining whether the difference between the measured real-world data and the model real-world data exceeds the failure threshold. The first autonomous function can be monitored (e.g., as described with respect to FIG. 9, the analysis module can output measured real-world data that can be compared with the model real-world data, and a difference between these two can be generated). The difference between these two can be compared with the failure threshold.

[0180] In 48210, if it is determined that a failure threshold has been exceeded, the surgical instrument may switch to the execution of a second autonomous function. The second autonomous function may be associated with a failure mitigation model. The failure mitigation model can generate failure mitigation feedback, and the failure mitigation feedback can result in adjusting parameters to values that enable the surgical instrument to perform a surgical task without risk. In an example, a difference between measured real-world data and model real-world data can be generated for each body measurement obtained from an expertise model. A body measurement related to the speed of a rotational actuator may be generated. This data may be processed by an analysis module. The measured real-world data related to the speed of the rotational actuator may be output from the analysis module.

[0181] Model data related to the speed of the rotational actuator may be included in the expertise model as a data structure such as a list. The speed of the rotational actuator from the model data structure may be compared to the actual speed of the rotational actuator within the measured real-world data, and each output may be generated based on the difference between these two, and each may be compared to a respective failure threshold. This may be performed for multiple body measurements that can generate respective measured real-world data that can be compared to respective model real-world data. The difference may be compared to each threshold (e.g., a failure threshold).

[0182] In an example, if one of the respective differences between the measured real-world data and the model real-world data exceeds a respective threshold, the surgical instrument may switch from a first autonomous function to a second autonomous function. The number of differences that must not be exceeded before switching to the second autonomous function may be manually input by the user of the surgical instrument. For example, a surgeon may set 3 as the number of times a respected difference must not exceed the failure threshold before switching to the second autonomous function.

[0183] The second autonomous function may be associated with a fault mitigation function as described herein. The difference between the measured real-world data and the modeled real-world data can be a single value. For example, an expertise model may include an analysis module that can evaluate the difference between each measured real-world data and each modeled real-world data and derive a value based on the difference between these two. In such a case, a single difference from all of the respective measured real-world data and modeled real-world data can be compared to a single fault threshold. When the single fault threshold is exceeded, the surgical instrument can switch from the first autonomous function to the second autonomous function to perform the surgical task.

[0184] A magnitude of the fault may be generated based on this difference. For example, the magnitude of the fault may be based on the extent to which the difference between the measured real-world data and the modeled real-world data exceeds the fault threshold. For example, if the difference between the measured real-world data and the modeled real-world data exceeds the fault threshold by a greater amount, the magnitude of the fault can be higher than when the difference exceeds the fault threshold by a smaller amount. The magnitude of the fault can be used when generating a fault mitigation parameter. For example, the fault mitigation parameter may be dynamic and may operate within a range. A higher magnitude of the fault can result in a surgical instrument using a parameter that poses less risk with respect to completion of the surgical task. In an example, the magnitude of the fault can be used to switch the surgical instrument from autonomous to manual.

[0185] In an example, the magnitude of the failure may be used to terminate all operations associated with the surgical task. The magnitude of the failure may be generated based on the surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or one or more of the degrees to which one or more of the outputs exceed the failure threshold. The output may mean the difference between the measured real data and the modeled real data. In an example, instead of generating the difference between the measured real data and the modeled real data, the measured real data may be compared to a threshold. From this comparison, it may be determined whether the threshold is exceeded, and in such a case, the surgical instrument may switch from the first autonomous function to the second autonomous function.

[0186] Weights may be assigned to the output of the analysis module (e.g., each of the outputs) and / or the measured real-world data (e.g., each of the measured real-world data). The speed of the actuator may be assigned a higher weight compared to the position of the sensor (e.g., potentiometer). Assigning weights may be performed locally in the surgical instrument or on the surgical hub by the analysis module. The assignment of weights may be performed on a remote service such as a cloud service. For example, the cloud service may be a lambda function of Amazon Web Services. Weights may be considered when comparing whether the difference between the modeled real-world data and the measured real-world data exceeds a failure threshold. For example, if the difference between the measured real-world data associated with the speed of the actuator exceeds the failure threshold, the surgical instrument may be more likely to switch to a second autonomous function. If the difference between the modeled real-world data and the measured real-world data associated with the position of the actuator exceeds the failure threshold, the surgical instrument may be less likely to switch to a second autonomous function. Assigning weights may be based on one or more of past data, surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs that exceed the failure threshold (e.g., the number of differences between the measured real-world data and the modeled real-world data that exceed the threshold), or the degree of difference of the outputs that exceed the failure threshold.

[0187] As described with respect to FIG. 9, when the surgical instrument switches from a first autonomous function associated with an expertise model to a second autonomous function associated with a fault reduction model, a message can be sent to the user of the surgical instrument. A set of recommendations may be included in the message to the user (e.g., the surgeon). For example, the recommendations may include which measured real-world data exceeded a threshold and may propose how the autonomous function can use fault reduction feedback to prevent any complications. In the example, the surgical instrument can switch back to a first autonomous function, which is an autonomous function associated with an expertise model. These switches can occur multiple times during the execution of the surgical task. These switches may be based on the difference between the measured real-world data and the modeled real-world data being within or below a threshold. In the example, a risk assessment model is included and can be used when determining whether to switch between autonomous functions.

[0188] Generating the recommendations may be based on the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the difference, the number of outputs that exceeded a fault threshold, and / or the degree to which an output exceeded a fault threshold. The expertise model may initialize parameters based on training data received before performing the surgical task. The training data may be based on past data, a surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs that exceeded a fault threshold, or one or more of the degree to which one or more outputs exceeded a threshold.

[0189] FIG. 12 shows an example of an overview of a surgical instrument 48215 having a control feedback 48220 and a fault reduction feedback 48221 and a surgical hub 48280. As shown with respect to FIG. 12, the surgical instrument 48215 can communicate with the surgical hub 48280. In the example, the surgical hub 48280 may be a third-party service such as an edge service on a cloud platform.

[0190] Surgical instrument 48215 can include control feedback 48220 and fault mitigation feedback 48221. Control feedback 48220 may be used while performing autonomous functions associated with a surgical task. For example, if the surgical task is mobilizing the colon, the autonomous function may be to free the colon from surrounding tissue, which may involve the use of an end cutter. Control feedback 48220 may set parameters for the end cutter to perform the autonomous function, e.g., to free the colon from surrounding tissue, via a message to the surgical instrument 48215, for example. Control feedback 48220 may adjust one or more parameters (e.g., autonomous parameters).

[0191] The parameters may include one or more actuators as described with respect to FIG. 9 and may be used as inputs for the actuators. The parameters may be adjusted based on feedback from a machine learning model. For example, an error rate may be measured based on the execution of the autonomous function, and if the error rate exceeds a threshold evaluated by the machine learning model, feedback may be sent to the surgical instrument to adjust one or more of the autonomous parameters.

[0192] Control feedback 48220 may include a memory. The memory may store one or more current parameters. The memory can store parameters via a database as described with respect to FIGS. 9 and 10. Control feedback 48220 may include a management module. The management module may be responsible for sending an update message to the memory when one or more of the autonomous parameters are updated. Control feedback may include a processor responsible for executing instructions to perform the autonomous function.

[0193] The processor can communicate with the memory. The memory can include an instruction memory that the processor can read from or write to. Control feedback 48220 can be associated with the normal operation of the autonomous function to perform an associated surgical task (e.g., autonomously). The control feedback can have one or more levels of autonomy. Switching the level of autonomy can result in adjusting one or more autonomous parameters. The level of autonomy can be stored in the memory along with one or more autonomous parameters associated with the level of autonomy.

[0194] Control feedback 48220 within the surgical instrument can generate an output. The output can be related to the execution of the autonomous function. The output can include one or more parameters of the autonomous function. The output can be generated and transmitted to the surgical hub 48280 via a message. The surgical hub 48280 can perform an analysis on the output. The surgical hub 48280 can transmit a response message. In such a case, the surgical instrument 48215 can receive input data from the surgical hub 48280 via the response message. The input data can include instructions for updating one or more autonomous parameters associated with the autonomous function.

[0195] The machine learning model can be disposed within the surgical hub 48280 as described herein. For example, the machine learning model can be a module within the surgical hub 48280. Storage 48305 can be included in the surgical hub. In an example, storage 48305 can be off - disk storage. The management module of the surgical hub 48280 can be responsible for transmitting data associated with the output data of the control feedback 48220 to the storage 48305. The output data can be associated with the execution of the autonomous function and / or the parameters of the autonomous function as described herein. Such data can be stored in the storage 48305 along with a timestamp when the output data was generated.

[0196] The management module of the surgical hub 48280 can send such data to the storage 48305. In the example, the analysis module may be included in the surgical hub 48280. The analysis module can compare the execution of the autonomous function generated by the output module with a threshold value (e.g., a fault reduction threshold value). The analysis module can determine whether the output data exceeds the fault reduction threshold value. In such a case, if such data exceeds the fault reduction threshold value, the surgical hub 48280 may send a message to the surgical instrument 48215 to switch from the use of the control feedback 48220 to the fault reduction feedback 48221. The fault reduction feedback 48221 can be the fault reduction feedback described with respect to FIGS. 9, 10, and 11.

[0197] The fault reduction feedback 48221 can include one or more actuators. The one or more actuators may be the same actuators as those described with respect to the control feedback 48220. The parameters used for the one or more actuators can be fault reduction parameters. The fault reduction parameters can be set to values where no risk or minimal risk is expected based on the execution of the autonomous function. For example, the fault reduction parameters can include values that result in a 100% success rate of the autonomous function. The fault reduction parameters can be stored in a memory associated with the fault reduction feedback 48221. The processor of the surgical instrument can request these values when executing the autonomous function. The fault reduction feedback can also be data generated by a fault reduction model (e.g., using a script associated with a second autonomous function), and this data is transmitted (e.g., signaled) to the hardware of the surgical instrument to autonomously control the surgical instrument when the surgical instrument performs a surgical task. The generated data can be set to values that ensure no risk or at least a reduction in risk is involved in the autonomous execution of the surgical task.

[0198] The failure mitigation parameter can be used as an input for the actuator to perform the surgical task at hand. The failure mitigation parameter can be transmitted from the surgical hub 48280. The performance of the surgical instrument 48215 using the failure mitigation parameter can generate output data associated with the performance, and the output data can be transmitted to and analyzed by the surgical hub 48280. The analysis module can ensure that there is no risk in the execution of the autonomous function.

[0199] The surgical hub 48280 can detect that risks still exist based on the performance of the failure mitigation parameter and / or the failure mitigation feedback output. The surgical hub 48280 can adjust the failure mitigation parameter that is transmitted via a message and received as an input by the surgical instrument 48215. The failure mitigation parameter can be updated in the memory. The management module associated with the failure mitigation feedback 48221 can be responsible for updating the memory. To execute the autonomous function using the failure mitigation feedback 48221, the processor can perform writing or reading to the memory associated with the failure mitigation feedback 48221.

[0200] Pre - monitoring and / or reaction may be based on encountered problems (e.g., automated problem solving). Autonomous function monitoring and / or fault mitigation may be provided, which may include monitoring of autonomous functions, magnitudes, and / or levels for the identification of faults in autonomous operations (e.g., part of an autonomous function). Identified faults can have a determination of the importance of the fault based on risk, redundancy, timing of occurrence, magnitude of occurrence, or impact of occurrence. Identified faults may be tracked by indication of the fault to a user and / or a system external to the hub system. The indication may, as a result, lead to active correction of the problem. The indication may be to a user, a facility, maintenance, a manufacturer, and / or a complaint database. The indication may result in fault mitigation, and the result of the mitigation may be part of an indication to one or more systems and / or users. Mitigation may, for example, reduce system performance to enable the end of a step or procedure. The affected part of the system can be shut down. It may include substitution of alternative means of monitoring and / or controlling medical hardware. It may, for example, enable the temporary halt and / or holding of instrument movement until the system fault is resolved.

[0201] A report may be provided, which may include the issuance of complaints and / or automated documentation. The automated documentation of complaints and / or issues may use the user guide form and / or template identification and assistant upon completion, complaint classification and / or routing (e.g., issues directed to the manufacturer and / or issues directed to the facility), machine learning aggregation and sorting for trends, or one or more of automated reports and / or summaries. The automated documentation of complaints and / or issues may include the surgical hub fully recognizing whether a device in the operating room was malfunctioning or in an error state. This information may be pulled automatically and pushed to the complaint database for review. The information can be classified in several ways. For example, the method may be to have the hub investigate the device records and determine the severity of the failure (e.g., DFMEA severity) and use this as a classification. One method may be to use the hospital's risk system as the basis for classification.

[0202] Faults, their potential causes, associated operating conditions, resolution or mitigation, and automated recording and / or reporting of user interactions or experiences may result from a fault. The system may record (e.g., automatically record) log data (e.g., all log data). The data can be stored in a database associated with the system. In an example, the system can record (e.g., automatically record) log data over a given period. For example, in the case of rollover logging, the system can log record (e.g., automatically log record) all data over a given period (e.g., regardless of other thresholds, severity, etc.). As time passes, the system can start automatically overwriting the oldest data first to conserve memory space. For the period of accountability, the system can, for example, automatically log record all data for a given period that may be legally required, and after that period has passed, can automatically discard the data. Identified data may be exempt from the automatic discard function. For patient outcomes, the log data may be retained from the time of surgery, for example, until the patient's outcome is confirmed. If a harmful event for that patient is identified, the data may be retained and not removed. If no harmful event is identified, the log data may be discarded after a specific period after surgery.

[0203] Automated post-operative actions backup and / or auditing can be used to review and / or document automated functions and their impact on users, surgical steps, and / or device results. User monitoring of reactions and responses to autonomous actions can be provided. An override of the primary response can be provided. For example, steps that a surgeon deviates from can be flagged for review. For example, steps within a surgery that the system considers to be at risk and the surgeon considers to be acceptable may be annotated as a user override, and the number of these triggers may be used as a trigger for auditing (e.g., AI review, peer review, deviation from IFU, etc.). The robotic system may execute the intended surgical steps and a device failure may occur. It may be determined that the robotic system is not malfunctioning, which may include failure logging and / or event logging. The end of the surgical audit of autonomous actions can be recorded based on the surgical review, user reactions, and / or monitored results to record success, failure, user selection, and / or the impact on subsequent steps. For example, automatic annotation of recorded images with a time overlay and device automation flags can be placed to provide context and situation awareness of the action.

[0204] A control system (e.g., a local facility and / or a global main system) can be warned about encountered problems. The intended use, instructions for use, and contraindications of the devices used during surgery may be provided. A medical device can indicate its intended use, instructions for use, and contraindications within the IFU. During the use of the device (e.g., each device), the system may identify whether a violation of the IFU is predicted to occur and may warn the user prior to the event. If the surgeon determines it is appropriate, a facility for overriding the system may be provided. For example, the control system may warn that the device should not be used on ischemic tissue near the colon. The surgeon's decision may be that the area is the best place for the progression. The surgeon can override the system and continue. The event may be recorded for analysis after the procedure. The number of firings of the device may be exceeded. It can be determined whether a battery-less material is used in any of the firings that can affect the total number of allowed firings. The procedure may have a total procedure time constraint. For example, if the device is set too long, the system can warn that the elapsed time has been exceeded. The surgeon can override if the decision to use is made. The tissue may not be accurately positioned and oriented within the joe. The placement and thickness of the tissue can be determined. The insertion of a counterfeit cartridge can be determined. It can be guaranteed that the device is cleaned before reloading the cartridge. The material of the device may be monitored to identify whether the patient has an allergy prior to use of the device and / or may warn the surgeon prior to the surgery. The problems may be aggregated into classifications. The classification may include the priority and / or importance of failures, user confusion, and / or hardware failures (e.g., product inquiries). There may be an escalation between classifications (e.g., frequency and / or severity). Low-risk hardware problems may have redundant systems in place. If the repeated use of the redundant system is initiated, it can be used as a trigger that the primary system is encountering problems.If the redundant system is only occasionally used, it may mean a necessary system or module reboot that can be regarded as the standard operation.

[0205] The hub and the attached system can provide self-service problem-solving. The automation of the image search function can be used within the surgical database for solution options. Images captured from the scope can bounce back from the database to identify healthy tissue versus diseased tissue as a confirmation before activating energy and / or stapling devices to minimize leakage and / or complications. In an example, it may be used to identify the optimal next step based on the problem and alert the surgeon. For example, if the surgeon completes the staple line and perfusion occurs, the surgeon may be warned of the best approach to address the leakage based on known past surgeries of the tissue type. For example, the hub may provide a recommendation (e.g., in the form of an SNS message) that suturing may be the best technique to address the leakage or that an energy modality that minimizes time provides the best technique. The images may be checked against the database for navigation confirmation. For example, when the scope is navigating to the intended site, it has a confirmation / verification check of the images from the scope that are compared to a known database to confirm that the surgeon is proceeding in the correct direction and can warn when off course. Network connection resolution can be provided. The magnitude of the failure can be automatically defined, and problems regarding functional operations and requirements on the system can be addressed using appropriate responses to experienced problems. For example, this can be used for failures in the operating room display. If the current operating room display is considered defective during a procedure, the system may transfer the content of the display to an alternative display. In an example, the system can split the information. The system can order defective components. Maintenance can call up a setup for the repair of the failed equipment. The system can alert problems for tracking. Low-risk hardware problems may have redundant systems in place. Major and minor failures and automatic responses by the hub may be provided. For example, the hub may experience problems regarding video output. A power supply to the system may fail, causing an overall loss of the video feed.This can be classified as a major failure state. An overlay system failure can include software errors. To ensure that the correct input is selected, the hub system can be capable of corresponding to various forms of output from the visualization system. The control means can ensure that the correct input (e.g., one having video execution) is selected for output. The display of the overlay may be stopped if the main fails (e.g., is unreliable). In such a case, it can be executed on a soft-core or hard-core redundant processor. An overlay system failure can include hardware failures. Redundant hardware may be used to maintain the full functionality of both systems and may include a voting system (e.g., similar to a fully redundant FCS). Redundant hardware can provide full video functionality without providing the overlay function. In the case of a switchover hardware solution, the system can have redundant power supplies. A solution that is difficult to achieve can result in an active solution (e.g., there are too many standards to switch).

[0206] An overlay system failure may include a power failure. In an example, a visualization system may be provided. The visualization system may fail. For example, the video may freeze (e.g., otherwise appear normal), which may be associated with the highest risk. The failure may be no video, ambiguous or interrupted video, monochrome video, video with color changes (e.g., lowest risk), etc. Video mitigation may be provided. For example, the control means may provide sufficient visualization to safely remove the instrument from the patient, and then a different visual platform may be found. This control means can include a device that can process the raw MIPI sensor output from the visualization system and render a Bayer pattern on a monitor. For example, a white light Bayer pattern sensor can output luminance values for each pixel (e.g., pixel) for each line and each frame, regardless of the color of the pixel. A series of red, green, and blue filters on individual pixels may filter the incident white light to provide intensity values of those colors. Image signal processing can use the Bayer pattern signal to calculate the missing color data at the pixel positions (e.g., each pixel position). This process is sometimes called demosaicing. By acquiring the raw sensor signal, placing it in memory, reading it from memory at the timing required by the HDMI display, and outputting it to the display, a preliminary monochrome image can be generated to enable safe removal of the instrument from the patient. The operation can be restored to its previous state using Quickboot or an archive of the operating conditions immediately before the failure or restart. System resource management can be provided.

[0207] The following is a numbered list of embodiments that may or may not be claimed.

[0208] 1. A device for controlling the execution of a surgical task by a surgical instrument, the device comprising a processor, the processor Monitor a first autonomous function associated with a surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task, Control a surgical instrument according to a first model based on the one or more tracked outputs, If at least one of the one or more tracked outputs exceeds a failure threshold, control the surgical instrument according to a failure mitigation model to switch the surgical instrument from performing a first autonomous function to performing a second autonomous function associated with the surgical task. A device configured as such.

[0209] Advantageously, these functions enable the combination of the device and the surgical instrument to function as a self - contained and fault - tolerant autonomous surgical system. Many instrument controllers that use any kind of output or feedback to control the instrument provide control commands as a simple function of the tracked output(s) (e.g., setting parameters as a linear function of the output value for the purpose of keeping the output constant, etc.). Thus, when a very abnormal or unexpected output is detected (typically associated with some kind of failure), the control commands provided to the instrument based on the output are also abnormal and are likely to be inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds the failure threshold. The device may be external to the surgical instrument or may be a processor or control circuit included within the surgical instrument.

[0210] 2. The device according to embodiment 1, wherein the first autonomous function and the second autonomous function are different autonomous functions.

[0211] This can be beneficial as it allows for a change in functionality when a failure is detected to enable the procedure to be safely paused, stopped, interrupted, or completed. For example, the first automated function may include a clamping operation, and when the threshold force is exceeded, the second autonomous function may be a jaw - opening operation.

[0212] 3. Controlling the surgical instrument according to the first model includes providing control feedback generated by the first model to the surgical instrument, where the control feedback is associated with setting and / or updating one or more parameters of the surgical instrument, and optionally, one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument. The device according to Embodiment 1 or Embodiment 2.

[0213] 4. The control feedback is configured to be used as an input for one or more actuators of the surgical instrument, and optionally, the control feedback is configured to set parameters via a message to the surgical instrument. The device according to Embodiment 3.

[0214] 5. One or more outputs associated with the first autonomous function and / or the surgical task include measurement values of sensors on the surgical instrument, and measurement values of wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task, optionally including heart rate, and measurement values of the positions of one or more actuators arranged on the surgical instrument performing the surgical task. The device according to any one of Embodiments 1 to 4 includes measured real-world data derived from one or more of these.

[0215] 6. Controlling the surgical instrument according to the first model based on the tracked output includes controlling the surgical instrument according to at least one output, or comparing at least one output with an ideal output generated by the first model to determine the difference between the two, and controlling the surgical instrument according to the difference. The device according to any one of Embodiments 1 to 5 includes either of these.

[0216] Determining whether at least one of the one or more outputs has exceeded a failure threshold includes determining that at least one output includes measured real-world data outside a given range, or comparing at least one output including measured real-world data to an ideal output generated by a first model to determine a difference between the two and determining that the difference is outside a given range, the device according to any one of embodiments 1 to 6 including any of these.

[0217] Determining whether at least one of the one or more outputs has exceeded a failure threshold includes determining whether a predetermined number of the one or more outputs each has exceeded the failure threshold, and optionally, the predetermined number is manually input by a user of the surgical instrument, the device according to any one of embodiments 1 to 7.

[0218] Controlling the surgical instrument according to a failure mitigation model includes generating or setting one or more parameters for the surgical instrument that are known or expected to reduce the risk associated with the completion of the surgical task, sending a message to the user of the surgical instrument, reducing the level of autonomy of the surgical instrument and / or switching the surgical instrument from autonomous settings to manual settings, ending all operations associated with the surgical task or reducing system performance to enable a safe end of the surgical task, the device according to any one of embodiments 1 to 8 including one or more of these.

[0219] Switching the surgical instrument from performing a first autonomous function to performing a second autonomous function associated with a surgical task can correspond to switching to less autonomous control, switching the operating parameters of the device (e.g., reducing the speed of the cutter, reversing the cutter, etc.), switching to an algorithm that stops the operation (e.g., switching from clamping to releasing), or changing the power level of the energy device, etc.

[0220] Advantageously, each of these fault mitigation responses can provide a response to a fault detected during the execution of a surgical task, which can reduce the risk of harm to the patient. For example, ending all operations related to the surgical task can stop the incorrect autonomous function within its track, preventing it from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation means (without simply ending the operation) can provide benefits over a simple fault mitigation control that simply stops the process within their track. This is because in various surgical contexts, better clinical outcomes (e.g., reducing harm to the patient or reducing complications) can be achieved by continuing the operation of the instrument in a safety-oriented manner rather than by ending its operation. For example, a surgical instrument having an end effector for clamping tissue, where the clamp arm is biased to the closed position, can be made safe by the controller by setting the actuator to unclamp the jaws of the end effector and thus release any tissue held therein.

[0221] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamp jaws of the end effector. During “normal” operation (i.e., operation according to the first model), this clamp force feedback may be used to control the speed of an actuator configured to advance the knife member and / or thread assembly. This can include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model can direct adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a high force is detected (indicating a larger, more robust tissue type) and slightly decreasing the firing speed when a low force is detected (indicating a smaller, more delicate tissue type). However, if it is determined that the force on the jaws has dropped below a failure threshold (e.g., has dropped completely to, or nearly to, zero), this may indicate that the tissue has slipped out of the jaws of the end effector. Accordingly, the system can switch to a failure mitigation model, which can direct setting the firing speed to a negative value (corresponding to reversing the direction of firing and driving the knife in the opposite direction within its recessed housing), as this is known (or at least reasonably foreseeable) to make the instrument safe by allowing repositioning of the end effector without significant risk to the patient or operator. The system can optionally also warn the user of the fact that this is occurring.

[0222] As another non-limiting example, when an unexpectedly strong force is detected to close the end effector of a surgical instrument, the system may switch to a failure mitigation model that can ramp up or ramp down the motor of the surgical instrument.

[0223] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply high-frequency (RF) and ultrasonic energy to tissue can be controlled according to a first model while the impedance level at the end effector electrode is being monitored. The first model may apply RF energy to the tissue and adjust the energy supply parameters (e.g., of the generator) based on the monitored impedance to ensure that the energy supply (e.g., power) at the surgical site remains substantially constant. However, if an unexpected spike in impedance (either very high or very low) is detected, this can exceed a fault threshold (defined as the range of impedance within which the detected impedance is expected to decrease during normal operation of the device). Thus, a switch can be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example can include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent unwanted cutting of the tissue.

[0224] 10. The one or more tracked outputs are the first one or more tracked outputs, The processor is further configured to monitor a second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task, A device according to any one of embodiments 1 to 9, wherein controlling the surgical instrument according to a fault mitigation model is based on the second one or more tracked outputs.

[0225] 11. The processor is further configured to generate a magnitude of a fault based on one or more outputs when one or more outputs exceed a fault threshold, and to control the surgical instrument according to a fault mitigation model, including adjusting one or more parameters of the surgical instrument based on the magnitude of the fault, for the device according to any one of embodiments 1 to 10.

[0226] Advantageously, this intelligently adjusts the response of the device and instrument to a fault to be proportional to the magnitude of that fault.

[0227] 12. The magnitude of the fault is generated based on at least one of a surgical context, a type of a first autonomous function, a type of a second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold, for the device according to embodiment 11.

[0228] Advantageously, this enables, for example, faults that are significant but potentially not numerous to be detected, and vice versa, and enables a balance to be taken between importance and quantity when the device mitigates the fault. Additionally or alternatively, it advantageously enables more important surgical functions (e.g., life and death) to be prioritized over less important surgical functions (e.g., utility / ease of use for the surgeon) for fault mitigation.

[0229] 13. The processor performs an analysis on one or more tracked outputs, is further configured to generate a comparison output based on the analysis, and the processor is configured to perform a switch from performing a first autonomous function to performing a second autonomous function when the comparison output exceeds the fault threshold, for the device according to any one of embodiments 1 to 12.

[0230] Advantageously, in this way, the switching of the autonomous function can be made in response to a relative output problem rather than an absolute output problem (e.g., a comparison can be made with an ideal, desired, and / or conceptual level of the output(s)). Additionally or alternatively, it may be beneficial to enable more nuances by making the switching of this function responsive to the relationship between one or more outputs.

[0231] 14. The device according to embodiment 13, wherein the processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of past data, surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or the degree to which one or more outputs exceed the failure threshold.

[0232] 15. The processor is further configured to transmit an indication of a failure message to a user of a surgical instrument associated with a surgical task when one or more outputs exceed a failure threshold, the failure message including a type of failure and a magnitude of the failure, for the device according to any one of embodiments 1 to 14.

[0233] 16. The device according to embodiment 15, wherein the failure message comprises a set of recommendations based on a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or the degree to which one or more outputs exceed the failure threshold.

[0234] 17. The device according to any one of embodiments 1 to 16, wherein the processor is further configured to determine a failure threshold and failure mitigation feedback based on training data.

[0235] 18. The device according to embodiment 17, wherein the training data is based on at least one of past data, surgical context, a first type of autonomous function, a second type of autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or the degree to which one or more outputs exceed the failure threshold.

[0236] 19. The device according to any one of embodiments 1 to 18, wherein the failure threshold is adjusted based on situation recognition.

[0237] 20. A computer-implemented method for controlling the execution of a surgical task by a surgical instrument, the method comprising: monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task; providing a digital command or user recommendation for controlling the surgical instrument according to a first model based on the one or more tracked outputs; providing a digital command or user recommendation for switching the surgical instrument from the execution of the first autonomous function to the execution of a second autonomous function associated with the surgical task by controlling the surgical instrument according to a failure mitigation model according to a determination that at least one of the one or more outputs exceeds a failure threshold.

[0238] Advantageously, these functions enable the combination of the computer implementing this method and the surgical instrument to function as a self - contained and fault - tolerant autonomous surgical system. Many instrument control methods that use any kind of output or feedback to control the instrument involve providing control instructions as a simple function of the tracked output(s) (e.g., setting parameters as a linear function of the output value for the purpose of keeping the output constant, etc.). Thus, when a very abnormal or unexpected output is detected (typically associated with some kind of fault), the control instructions provided to the instrument based on the output are also likely to be abnormal and inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds a fault threshold. The computer implementing this method may be external to the surgical instrument or may be a processor or control circuit included within the surgical instrument itself.

[0239] 21. The method according to embodiment 20, wherein the first autonomous function and the second autonomous function are different autonomous functions.

[0240] This can be beneficial because it allows the function to be changed when a fault is detected in order to enable the procedure to be safely paused, stopped, interrupted, or completed. For example, the first automatic function may include a clamping operation, and when a threshold force is exceeded, the second autonomous function may be a jaw - opening operation.

[0241] 22. The digital command or user recommendation for controlling the surgical instrument according to the first model includes one or more values for setting or updating the parameters of the surgical instrument, and optionally, one or more parameters of the surgical instrument include the parameters of a script for controlling the surgical instrument, the method according to embodiment 20 or embodiment 21.

[0242] 23. The method according to embodiment 22, wherein the value is suitable for use as an input for one or more actuators of a surgical instrument, and optionally, the digital command is a message to a surgical instrument configured to set parameters of the surgical instrument.

[0243] 24. One or more outputs associated with a first autonomous function and / or a surgical task are measurement values of sensors on the surgical instrument, and measurement values of wearable sensors that measure patient or surgeon biomarkers or other health markers associated with the surgical task, optionally including heart rate, and measurement values of the positions of one or more actuators arranged on the surgical instrument that performs the surgical task, and include measured real-world data derived from one or more of these, according to any one of embodiments 20 to 23.

[0244] 25. A digital command or user recommendation for controlling the surgical instrument according to a first model based on the tracked output is a digital command or user recommendation for controlling the surgical instrument according to at least one output, or a digital command or user recommendation for controlling the surgical instrument according to a difference determined by comparing at least one output with an ideal output generated by the first model, according to any one of embodiments 20 to 24.

[0245] 26. Determining whether at least one of the one or more outputs has exceeded a failure threshold is determining that the measured real-world data includes at least one output outside a given range, or comparing at least one output including the measured real-world data with an ideal output generated by the first model to determine the difference between the two, and determining that the difference is outside a given range, according to any one of embodiments 20 to 25.

[0246] Determining whether at least one of one or more outputs has exceeded a fault threshold includes determining whether a predetermined number of the one or more outputs each has exceeded the fault threshold, and optionally, the predetermined number is manually input by a user of the surgical instrument, the method according to any one of embodiments 20 to 26.

[0247] 28. A digital command or user recommendation for switching a surgical instrument from performing a first autonomous function to performing a second autonomous function is for generating or setting one or more parameters for the surgical instrument that are known or expected to reduce the risk associated with the completion of a surgical task, for sending a message to a user of the surgical instrument, for reducing the level of autonomy of the surgical instrument and / or for switching the surgical instrument from an autonomous setting to a manual setting, for ending all operations associated with a surgical task, or or includes a digital command or user recommendation for reducing system performance to enable a safe end of a surgical task, the method according to any one of embodiments 20 to 27.

[0248] Switching a surgical instrument from performing a first autonomous function to performing a second autonomous function associated with a surgical task can correspond to a switch to less autonomous control, a switch of the operating parameters of the device (e.g., reduction of the cutter speed, reversal of the cutter, etc.), a switch to an algorithm for stopping the operation (e.g., switch from clamping to releasing), or a change in the power level of an energy device.

[0249] Advantageously, each of these failure mitigation responses can provide a response to a failure detected during the execution of a surgical task, which can reduce the risk of harm to the patient. For example, terminating all operations associated with the surgical task can stop the incorrect autonomous function within its track, preventing it from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other failure mitigation means (without simply terminating the operation) can provide benefits over a simple failure mitigation control that simply stops the process within their track. This is because in various surgical contexts, better clinical outcomes (e.g., reduction of harm to the patient or reduction of complications) can be achieved by continuing the operation of the instrument in a safety-oriented manner rather than by terminating its operation. For example, a surgical instrument having an end effector for clamping tissue, where the clamp arm is biased to a closed position, can be made safe by the controller by setting the actuator to unclamp the jaws of the end effector and thus release any tissue held therein.

[0250] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamp jaws of the end effector. During “normal” operation (i.e., operation according to the first model), this clamp force feedback may be used to control the speed of an actuator configured to advance the knife member and / or the thread assembly. This can include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model can direct adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a high force is detected (indicating a larger, more robust tissue type) and slightly decreasing the firing speed when a low force is detected (indicating a smaller, more delicate tissue type). However, if it is determined that the force on the jaws has dropped below a failure threshold (e.g., has dropped completely to, or nearly to, zero), this may indicate that the tissue has slipped out of the jaws of the end effector. Thus, the system can switch to a failure mitigation model, which can direct setting the firing speed to a negative value (corresponding to reversing the direction of firing and driving the knife in the opposite direction within its recessed housing), as this is known (or at least reasonably foreseeable) to make the instrument safe by allowing repositioning of the end effector without significant risk to the patient or operator. The system can optionally also warn the user of the fact that this is occurring.

[0251] As another non-limiting example, when an unexpectedly strong force is detected to close the end effector of a surgical instrument, the system may switch to a failure mitigation model that can ramp up or ramp down the motor of the surgical instrument.

[0252] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply high-frequency (RF) and ultrasonic energy to tissue can be controlled according to a first model while the impedance level at the end effector electrode is being monitored. The first model may apply RF energy to the tissue and adjust the energy supply parameters (e.g., of the generator) based on the monitored impedance to ensure that the energy supply (e.g., power) at the surgical site remains substantially constant. However, if an unexpected spike in impedance (either very high or very low) is detected, this can exceed a fault threshold (defined as the range of impedance within which the detected impedance is expected to decrease during normal operation of the device). Thus, a switch can be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example can include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent unwanted cutting of the tissue.

[0253] 29. The one or more tracked outputs are the first one or more tracked outputs, The method further includes monitoring a second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task, Digital commands or user recommendations for controlling a surgical instrument according to a fault mitigation model are the method according to any one of embodiments 20 to 28, based on the second one or more tracked outputs.

[0254] 30. If one or more outputs exceed a failure threshold, further comprising generating a magnitude of the failure based on the one or more outputs, wherein the digital command or user recommendation is for adjusting one or more parameters of the surgical instrument based on the magnitude of the failure, the method according to any one of embodiments 20 to 29.

[0255] Advantageously, this intelligently adjusts the response of the computer and the instrument to the failure to be proportional to the magnitude of that failure.

[0256] 31. The magnitude of the failure is generated based on at least one of a surgical context, a type of a first autonomous function, a type of a second autonomous function, past data associated with the first autonomous function, a number of outputs that exceeded the failure threshold, or a degree to which one or more outputs exceeded the failure threshold, the method according to embodiment 30.

[0257] Advantageously, this allows, for example, for the detection of failures that may be significant but not numerous, and vice versa, and allows the computer to balance importance and quantity when reducing the failure. Additionally or alternatively, it advantageously allows for more important surgical functions (e.g., life and death) to be prioritized over less important surgical functions (e.g., utility / ease of use for the surgeon) for failure reduction.

[0258] 32. performing an analysis on the one or more outputs being tracked; generating a comparison output based on the analysis; and providing a digital command or user recommendation for switching the surgical instrument from performing a first autonomous function to performing a second autonomous function according to a determination that the comparison output exceeded the failure threshold, the method according to any one of embodiments 20 to 31.

[0259] Advantageously, in this way, the switching of the autonomous function can be made in response to a relative output problem rather than an absolute output problem (e.g., a comparison can be made with an ideal, desired, and / or conceptual level of the output(s)). Additionally or alternatively, it may be beneficial to make the switching of this function responsive to the relationship between one or more outputs to allow for more nuances.

[0260] 33. The method of embodiment 32, further comprising assigning a weight to each of the one or more tracked outputs based on at least one of past data, surgical context, a first type of autonomous function, a second type of autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a fault threshold, or the degree to which one or more outputs exceed a fault threshold.

[0261] 34. The method according to any one of embodiments 20 to 33, further comprising sending an indication of a fault message to a user of a surgical instrument associated with a surgical task when one or more outputs exceed a fault threshold, the fault message including a fault type and a magnitude of the fault.

[0262] 35. The method of embodiment 34, wherein the fault message comprises a set of recommendations based on a first type of autonomous function, a second type of autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a fault threshold, or the degree to which one or more outputs exceed a fault threshold.

[0263] 36. The method according to any one of embodiments 20 to 35, further comprising determining a fault threshold and fault mitigation feedback based on training data.

[0264] 37. The method according to embodiment 36, wherein the training data is based on at least one of past data, surgical context, a first type of autonomous function, a second type of autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a failure threshold, or the degree to which one or more outputs exceed the failure threshold.

[0265] 38. The method according to any one of embodiments 20 to 37, wherein the failure threshold is adjusted based on situation recognition.

[0266] 39. A computer program product for causing a computer to execute the method according to any one of embodiments 20 to 38.

[0267] 40. A non-transitory computer-readable storage medium including computer-readable instructions that, when executed by a computer, cause the computer to execute the method according to any one of embodiments 20 to 38.

[0268] The following are the numbered aspects of the present disclosure, which may or may not be claimed.

[0269] 1. A device for monitoring a first autonomous function associated with a surgical task, comprising a processor, the processor executes a first autonomous function associated with a surgical task, and executing the first autonomous function is associated with control feedback, and monitors the first autonomous function by tracking one or more outputs associated with the control feedback, when one or more outputs exceed a failure threshold, switches from execution of the first autonomous function to execution of a second autonomous function associated with the surgical task, and the second autonomous function is configured to be associated with failure mitigation feedback.

[0270] 2. The processor When one or more outputs exceed a failure threshold, generate a magnitude of failure based on the one or more outputs, The device according to aspect 1, further configured to adjust one or more parameters associated with failure mitigation feedback based on the magnitude of the failure.

[0271] 3. The magnitude of the failure is generated based on at least one of a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs that exceeded the failure threshold, or a degree to which one or more outputs exceeded the failure threshold, of the device according to aspect 2.

[0272] 4. The processor performs an analysis on one or more tracked outputs associated with control feedback, generates a comparison output based on the analysis, When the comparison output exceeds a failure threshold, switch from the execution of the first autonomous function to the execution of a second autonomous function associated with a surgical task, and the second autonomous function is further configured to be associated with failure mitigation feedback, of the device according to aspect 1.

[0273] 5. The processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of past data, a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs that exceeded the failure threshold, or a degree to which one or more outputs exceeded the failure threshold, of the device according to aspect 4.

[0274] 6. The processor is further configured to send an indication of a failure message to a user of a surgical instrument associated with a surgical task when one or more outputs exceed a failure threshold, and the failure message includes a type of failure and a magnitude of the failure, of the device according to aspect 1.

[0275] 7. The fault message includes a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, the past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold, for the device according to aspect 6.

[0276] 8. The processor is further configured to determine a fault threshold and a fault mitigation feedback based on training data, for the device according to aspect 1.

[0277] 9. The training data is based on at least one of past data, surgical context, the type of the first autonomous function, the type of the second autonomous function, the past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold, for the device according to aspect 8.

[0278] 10. The fault threshold is adjusted based on situation awareness, for the device according to aspect 1.

[0279] 11. A method for monitoring a first autonomous function associated with a surgical task, executing a first autonomous function associated with a surgical task, wherein executing the first autonomous function is associated with control feedback, monitoring the first autonomous function by tracking one or more outputs associated with the control feedback, when one or more outputs exceed a fault threshold, switching from executing the first autonomous function to executing a second autonomous function associated with the surgical task, wherein the second autonomous function is associated with fault mitigation feedback,

[0280] 12. when one or more outputs exceed a fault threshold, generating a magnitude of the fault based on the one or more outputs Further comprising adjusting one or more parameters associated with fault mitigation feedback based on the magnitude of the fault, the method according to aspect 11.

[0281] 13. The magnitude of the fault is generated based on at least one of a surgical context, a type of a first autonomous function, a type of a second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a fault threshold, or the degree to which one or more outputs exceed the fault threshold, the method according to aspect 12.

[0282] 14. Performing an analysis on one or more tracked outputs associated with control feedback; Generating a comparison output based on the analysis; Switching from the execution of the first autonomous function to the execution of a second autonomous function associated with a surgical task when the comparison output exceeds a fault threshold, wherein the second autonomous function is associated with fault mitigation feedback, the method according to aspect 11.

[0283] 15. Further comprising assigning a weight to each of the one or more tracked outputs based on at least one of past data, a surgical context, a type of a first autonomous function, a type of a second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding a fault threshold, or the degree to which one or more outputs exceed the fault threshold, the method according to aspect 14.

[0284] 16. Further comprising transmitting an indication of a fault message to a user of a surgical instrument associated with a surgical task when one or more outputs exceed a fault threshold, the fault message including a fault type and a magnitude of the fault, the method according to aspect 11.

[0285] 17. The fault message includes a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, the past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold, in the method according to aspect 16.

[0286] 18. The method according to aspect 11, further comprising determining a fault threshold and a fault mitigation feedback based on training data.

[0287] 19. The training data is based on at least one of past data, surgical context, the type of the first autonomous function, the type of the second autonomous function, the past data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which one or more outputs exceed the fault threshold, in the method according to aspect 18.

[0288] 20. The fault threshold is adjusted based on situation recognition, in the method according to aspect 11.

[0289] 〔Embodiment〕 (1) A device for controlling the execution of a surgical task by a surgical instrument, the device comprising a processor, the processor monitoring a first autonomous function associated with the surgical task by tracking the first autonomous function and / or one or more outputs associated with the first autonomous function and / or the surgical task, controlling the surgical instrument according to a first model based on the one or more tracked outputs, configured to switch the surgical instrument from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model when at least one of the one or more tracked outputs exceeds a fault threshold. (2) The device according to embodiment 1, wherein the first autonomous function and the second autonomous function are different autonomous functions. (3) Controlling the surgical instrument according to the first model includes providing control feedback generated by the first model to the surgical instrument, the control feedback being associated with setting and / or updating one or more parameters of the surgical instrument, and optionally, the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument, the device according to embodiment 1 or embodiment 2. (4) The control feedback is configured to be used as an input for one or more actuators of the surgical instrument, and optionally, the control feedback is configured to set the parameters via a message to the surgical instrument, the device according to embodiment 3. (5) The one or more outputs associated with the first autonomous function and / or the surgical task are measurements of sensors on the surgical instrument, and optionally measurements of wearable sensors that measure biomarkers or other health markers of a patient or surgeon associated with the surgical task, including heart rate, and measurements of the positions of one or more actuators disposed on the surgical instrument performing the surgical task, and include measured real-world data derived from one or more of these, the device according to any of embodiments 1 to 4.

[0290] (6) Controlling the surgical instrument according to the first model based on the tracked output is controlling the surgical instrument according to at least one output, or comparing at least one output with an ideal output generated by the first model to determine the difference between the two, and controlling the surgical instrument according to the difference, and includes either of these, the device according to any of embodiments 1 to 5. (7) Determining whether at least one of the one or more outputs has exceeded a failure threshold is Determining that the measured real-world data includes at least one output outside a given range, or Comparing at least one output including the measured real-world data with an ideal output generated by the first model to determine a difference between the two, and determining that the difference is outside a given range, the device according to any one of Embodiments 1 to 6 including either of these. (8) Determining whether at least one of the one or more outputs has exceeded a failure threshold includes determining whether a predetermined number of the one or more outputs have each exceeded the failure threshold, and optionally, the predetermined number is manually input by a user of the surgical instrument, the device according to any one of Embodiments 1 to 7. (9) Controlling the surgical instrument according to a failure mitigation model Generating or setting one or more parameters for the surgical instrument, known or expected to reduce the risk associated with completion of the surgical task Sending a message to the user of the surgical instrument Reducing the autonomy level of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting Ending all operations associated with the surgical task, or Reducing system performance to enable a safe end of the surgical task, the device according to any one of Embodiments 1 to 8 including one or more of these. (10) The one or more tracked outputs are a first one or more tracked outputs The processor is further configured to monitor the second autonomy function associated with the surgical task by tracking the second autonomy function and / or a second one or more outputs associated with the surgical task Controlling the surgical instrument according to the failure mitigation model is based on the second one or more tracked outputs, the device according to any one of Embodiments 1 to 9.

[0291] (11) The device according to any one of embodiments 1 to 10, wherein when one or more of the outputs exceed the failure threshold, the processor is further configured to generate a magnitude of the failure based on the one or more outputs, and controlling the surgical instrument according to the failure mitigation model includes adjusting one or more parameters of the surgical instrument based on the magnitude of the failure. (12) The device according to embodiment 11, wherein the magnitude of the failure is generated based on at least one of a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs exceeding the failure threshold, or a degree to which the one or more outputs exceed the failure threshold. (13) The processor performs an analysis on the one or more tracked outputs, and is further configured to generate a comparison output based on the analysis, and the processor is configured to perform a switch from performing the first autonomous function to performing the second autonomous function when the comparison output exceeds the failure threshold. The device according to any one of embodiments 1 to 12. (14) The device according to embodiment 13, wherein the processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of past data, a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs exceeding the failure threshold, or a degree to which the one or more outputs exceed the failure threshold. (15) The processor is further configured to send an indication of a failure message to a user of the surgical instrument associated with the surgical task when the one or more outputs exceed the failure threshold, and the failure message includes a type of failure and a magnitude of the failure. The device according to any one of embodiments 1 to 14.

[0292] (16) The device according to embodiment 15, wherein the failure message comprises a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which the one or more outputs exceed the failure threshold. (17) The device according to any one of embodiments 1 to 16, wherein the processor is further configured to determine the failure threshold and failure mitigation feedback based on training data. (18) The device according to embodiment 17, wherein the training data is based on at least one of past data, surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which the one or more outputs exceed the failure threshold. (19) The device according to any one of embodiments 1 to 18, wherein the failure threshold is adjusted based on situation recognition. (20) A computer-implemented method for controlling the execution of a surgical task by a surgical instrument, the method comprising: monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task; providing a digital command or user recommendation for controlling the surgical instrument according to a first model based on the one or more tracked outputs; providing a digital command or user recommendation for switching the surgical instrument from the execution of the first autonomous function to the execution of a second autonomous function associated with the surgical task by controlling the surgical instrument according to a failure mitigation model according to a determination that at least one of the one or more outputs exceeds a failure threshold.

[0293] (21) The method according to embodiment 20, wherein the first autonomous function and the second autonomous function are different autonomous functions. (22) The digital command or user recommendation for controlling the surgical instrument according to the first model includes one or more values for setting or updating parameters of the surgical instrument, and optionally, the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument. The method according to embodiment 20 or embodiment 21. (23) The value is suitable for use as an input for one or more actuators of the surgical instrument, and optionally, the digital command is a message to the surgical instrument configured to set the parameters of the surgical instrument. The method according to embodiment 22. (24) The one or more outputs associated with the first autonomous function and / or the surgical task are measurement values of sensors on the surgical instrument, and measurement values of wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task, optionally including heart rate, and measurement values of the positions of one or more actuators arranged on the surgical instrument for performing the surgical task. The method according to any one of embodiments 20 to 23, including measured real data derived from one or more of them. (25) The digital command or user recommendation for controlling the surgical instrument according to the first model based on the tracked output is a digital command or user recommendation for controlling the surgical instrument according to at least one output, or a digital command or user recommendation for controlling the surgical instrument according to a difference determined by comparing at least one output with an ideal output generated by the first model. The method according to any one of embodiments 20 to 24.

[0294] Determining whether at least one of the one or more outputs has exceeded a failure threshold comprises determining that at least one output includes measured real-world data that is outside a given range, or comparing at least one output that includes measured real-world data to an ideal output generated by the first model to determine a difference between the two and determining that the difference is outside a given range, the method according to any of embodiments 20 to 25, including either. (27) Determining whether at least one of the one or more outputs has exceeded a failure threshold includes determining whether a predetermined number of the one or more outputs each has exceeded the failure threshold, and optionally, the predetermined number is manually input by a user of the surgical instrument, the method according to any of embodiments 20 to 26. (28) The digital command or user recommendation for switching the surgical instrument from execution of the first autonomous function to execution of the second autonomous function is for generating or setting one or more parameters for the surgical instrument, known or expected to reduce the risk associated with completion of the surgical task, for sending a message to the user of the surgical instrument, for reducing the level of autonomy of the surgical instrument and / or for switching the surgical instrument from an autonomous setting to a manual setting, for ending all operations associated with the surgical task, or for reducing system performance to enable a safe end of the surgical task, the method according to any of embodiments 20 to 27, including a digital command or user recommendation. (29) The one or more tracked outputs are the first one or more tracked outputs, The method further includes monitoring the second autonomous function associated with the surgical task by tracking the second autonomous function and / or one or more second outputs associated with the surgical task, The digital command or user recommendation for controlling the surgical instrument according to the fault mitigation model is the method according to any one of embodiments 20 to 28, based on the one or more second tracked outputs. (30) When the one or more outputs exceed the fault threshold, further including generating a magnitude of the fault based on the one or more outputs, wherein the digital command or user recommendation is for adjusting one or more parameters of the surgical instrument based on the magnitude of the fault, the method according to any one of embodiments 20 to 29.

[0295] (31) The magnitude of the fault is generated based on at least one of a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs exceeding the fault threshold, or a degree to which the one or more outputs exceed the fault threshold, the method according to embodiment 30. (32) further including performing an analysis on the one or more tracked outputs, generating a comparison output based on the analysis, and providing the digital command or user recommendation for switching the surgical instrument from execution of the first autonomous function to execution of the second autonomous function according to a determination that the comparison output exceeds the fault threshold, the method according to any one of embodiments 20 to 31. (33) further including assigning a weight to each of the one or more tracked outputs based on at least one of past data, a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs exceeding the fault threshold, or a degree to which the one or more outputs exceed the fault threshold, the method according to embodiment 32. (34) Further comprising, when one or more of the outputs exceed the failure threshold, sending an indication of a failure message to a user of a surgical instrument associated with the surgical task, the failure message including a failure type and a magnitude of the failure, the method according to any one of embodiments 20 to 33. (35) The failure message comprises a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which one or more of the outputs exceed the failure threshold, the method according to embodiment 34.

[0296] (36) Further comprising determining the failure threshold and failure mitigation feedback based on training data, the method according to any one of embodiments 20 to 35. (37) The training data is based on at least one of past data, a surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which one or more of the outputs exceed the failure threshold, the method according to embodiment 36. (38) The failure threshold is adjusted based on situation awareness, the method according to any one of embodiments 20 to 37. (39) A computer program product for causing a computer to execute the method according to any one of embodiments 20 to 38. (40) A non-transitory computer-readable storage medium including computer-readable instructions that, when executed by a computer, cause the computer to execute the method according to any one of embodiments 20 to 38.

Claims

1. A device for controlling the execution of a surgical task by a surgical instrument, the device comprising a processor, the processor monitoring a first autonomous function associated with the surgical task by tracking the first autonomous function and / or one or more outputs associated with the surgical task, controlling the surgical instrument according to a first model based on the one or more tracked outputs, and configured to switch the surgical instrument from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model when at least one of the one or more tracked outputs exceeds a fault threshold.

2. The device according to claim 1, wherein the first autonomous function and the second autonomous function are different autonomous functions.

3. Controlling the surgical instrument according to the first model includes providing control feedback generated by the first model to the surgical instrument, the control feedback being associated with setting and / or updating one or more parameters of the surgical instrument, and optionally, the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument. The device according to claim 1 or claim 2.

4. The control feedback is configured to be used as an input for one or more actuators of the surgical instrument, and optionally, the control feedback is configured to set the parameters via a message to the surgical instrument. The device according to claim 3.

5. The one or more outputs associated with the first autonomous function and / or the surgical task are measurements of sensors on the surgical instrument, and Measurements from a wearable sensor that measures patient or surgeon biomarkers or other health markers associated with the surgical task, optionally including heart rate, and Measured real-world data derived from one or more of: measurements of the position of one or more actuators disposed on the surgical instrument that performs the surgical task, the device of claim 1. **Claim 6** Controlling the surgical instrument according to the first model based on the tracked output, Controlling the surgical instrument according to at least one output, or Comparing at least one output to an ideal output generated by the first model, determining the difference between the two, and controlling the surgical instrument according to the difference, the device of claim 1. **Claim 7** Determining whether at least one of the one or more outputs has exceeded a fault threshold, Determining that the measured real-world data includes at least one output outside a given range, or Comparing at least one output, including measured real-world data, to an ideal output generated by the first model, determining the difference between the two, and determining that the difference is outside a given range, the device according to any one of claims 1 to 6. **Claim 8** Determining whether at least one of the one or more outputs has exceeded a fault threshold includes determining whether a predetermined number of the one or more outputs each exceed the fault threshold, and optionally, the predetermined number is manually input by a user of the surgical instrument, the device of claim 1. **Claim 9** Controlling the surgical instrument according to a fault mitigation model, Generating or setting one or more parameters for the surgical instrument that are known or expected to reduce the risk associated with completion of the surgical task, Sending a message to the user of the surgical instrument, Reducing the level of autonomy of the surgical instrument and / or switching the surgical instrument from autonomous settings to manual settings, Ending all operations related to the surgical task, or, Reducing system performance to enable a safe end to the surgical task, one or more of which are included in the device of claim 1.

10. The one or more tracked outputs are the first one or more tracked outputs, The processor is further configured to monitor the second autonomous function associated with the surgical task by tracking the second autonomous function and / or a second one or more outputs associated with the surgical task, Controlling the surgical instrument according to the fault mitigation model is based on the second one or more tracked outputs, the device of claim 1.

11. The processor is further configured to generate a magnitude of the fault based on the one or more outputs when the one or more outputs exceed the fault threshold, and controlling the surgical instrument according to the fault mitigation model includes adjusting one or more parameters of the surgical instrument based on the magnitude of the fault, the device of claim 1.

12. The magnitude of the fault is generated based on at least one of a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs that exceeded the fault threshold, or a degree to which the one or more outputs exceeded the fault threshold, the device of claim 11.

13. The processor is, perform analysis on the one or more tracked outputs, and is further configured to generate a comparison output based on the analysis, wherein the processor is configured to perform a switch from executing the first autonomous function to executing the second autonomous function when the comparison output exceeds the failure threshold value. The device according to claim 1.

14. The processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of past data, surgical context, type of the first autonomous function, type of the second autonomous function, past data associated with the first autonomous function, number of outputs exceeding the failure threshold value, or degree to which the one or more outputs exceed the failure threshold value. The device according to claim 13.

15. when the one or more outputs exceed the failure threshold value, the processor is further configured to send an indication of a failure message to a user of a surgical instrument associated with the surgical task, and the failure message includes a failure type and a magnitude of the failure. The device according to claim 1.

16. The failure message according to claim 15 includes a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold value, or the degree to which the one or more outputs exceed the failure threshold value. The device according to claim 15.

17. The processor is further configured to determine the failure threshold value and failure mitigation feedback based on training data. The device according to claim 1.

18. The device according to claim 17, wherein the training data is based on at least one of past data, surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which the one or more outputs exceed the failure threshold.

19. The device according to claim 1, wherein the failure threshold is adjusted based on situation recognition.

20. A computer-implemented method for controlling the execution of a surgical task by a surgical instrument, the method comprising: monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task; providing a digital command or user recommendation for controlling the surgical instrument according to a first model based on the one or more tracked outputs; providing a digital command or user recommendation for switching the surgical instrument from the execution of the first autonomous function to the execution of a second autonomous function associated with the surgical task by controlling the surgical instrument according to a failure mitigation model according to a determination that at least one of the one or more outputs exceeds a failure threshold.

21. The method according to claim 20, wherein the first autonomous function and the second autonomous function are different autonomous functions.

22. The digital command or user recommendation for controlling the surgical instrument according to the first model includes one or more values for setting or updating parameters of the surgical instrument, and optionally, the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument. The method according to claim 20 or claim 21.

23. The method according to claim 22, wherein the value is suitable for use as an input for one or more actuators of the surgical instrument, and optionally, the digital command is a message to the surgical instrument configured to set the parameters of the surgical instrument.

24. The one or more outputs associated with the first autonomous function and / or the surgical task are measurement values of sensors on the surgical instrument, and measurement values of wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task, optionally including heart rate, and measurement values of the positions of one or more actuators arranged on the surgical instrument for performing the surgical task, and the measured real-world data derived from one or more of them, the method according to claim 20.

25. The digital command or user recommendation for controlling the surgical instrument according to a first model based on the tracked output is a digital command or user recommendation for controlling the surgical instrument according to at least one output, or a digital command or user recommendation for controlling the surgical instrument according to a difference determined by comparing at least one output with an ideal output generated by the first model, the method according to claim 20.

26. Determining whether at least one of the one or more outputs has exceeded a failure threshold is determining that the measured real-world data includes at least one output outside a given range, or comparing at least one output including the measured real-world data with an ideal output generated by the first model to determine the difference between the two, and determining that the difference is outside a given range, the method according to claim 20.

27. Determining whether at least one of the one or more outputs has exceeded a failure threshold includes determining whether a predetermined number of the one or more outputs have each exceeded the failure threshold, and optionally, the predetermined number is manually input by a user of the surgical instrument, the method of claim 20. **Claim 28** The digital command or user recommendation for switching the surgical instrument from execution of the first autonomous function to execution of the second autonomous function is For generating or setting one or more parameters for the surgical instrument, known or expected to reduce the risk associated with completion of the surgical task For sending a message to the user of the surgical instrument For reducing the level of autonomy of the surgical instrument and / or for switching the surgical instrument from autonomous settings to manual settings For ending all operations associated with the surgical task, or Including a digital command or user recommendation for reducing system performance to enable a safe end of the surgical task, the method of claim 20. **Claim 29** The one or more tracked outputs are a first one or more tracked outputs, The method further includes monitoring the second autonomous function associated with the surgical task by tracking the second autonomous function and / or a second one or more outputs associated with the surgical task, The digital command or user recommendation for controlling the surgical instrument according to the failure mitigation model is based on the second one or more tracked outputs, the method of claim 20. **Claim 30** If the one or more outputs exceed the failure threshold, further comprising generating a magnitude of the failure based on the one or more outputs, wherein the digital command or user recommendation is for adjusting one or more parameters of the surgical instrument based on the magnitude of the failure, the method according to claim 20.

31. The magnitude of the failure is generated based on at least one of a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs exceeding the failure threshold, or a degree to which the one or more outputs exceed the failure threshold, the method according to claim 30.

32. performing an analysis on the one or more outputs being tracked; generating a comparison output based on the analysis, and in accordance with a determination that the comparison output exceeds the failure threshold, the digital command or user recommendation for switching the surgical instrument from execution of the first autonomous function to execution of a second autonomous function is provided, the method according to claim 20.

33. further comprising assigning a weight to each of the one or more outputs being tracked based on at least one of past data, a surgical context, a type of the first autonomous function, a type of the second autonomous function, past data associated with the first autonomous function, a number of outputs exceeding the failure threshold, or a degree to which the one or more outputs exceed the failure threshold, the method according to claim 32.

34. If the one or more outputs exceed the failure threshold, further comprising transmitting an indication of a failure message to a user of the surgical instrument associated with the surgical task, the failure message including a type of failure and a magnitude of the failure, the method according to claim 20.

35. The method according to claim 34, wherein the failure message comprises a set of recommendations based on the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which the one or more outputs exceed the failure threshold.

36. The method according to claim 20, further comprising determining the failure threshold and failure mitigation feedback based on training data.

37. The method according to claim 36, wherein the training data is based on at least one of past data, surgical context, the type of the first autonomous function, the type of the second autonomous function, past data associated with the first autonomous function, the number of outputs exceeding the failure threshold, or the degree to which the one or more outputs exceed the failure threshold.

38. The method according to claim 20, wherein the failure threshold is adjusted based on situation awareness.

39. A computer program product for causing a computer to execute the method according to claim 20.

40. A non-transitory computer-readable storage medium including computer-readable instructions that, when executed by a computer, cause the computer to execute the method according to claim 20.

Images