Detecting fault mitigation associated with autonomous surgical tasks

Abstract

Systems, methods, and instruments for detecting failure mitigation associated with a surgical task are described herein. The device may perform a first autonomous function associated with a surgical task. Performing the first autonomous function may be associated with control feedback. The first autonomous function may be monitored by tracking one or more outputs associated with the control feedback. If the one or more outputs exceed a fault threshold, the device may switch from performing the first autonomous function to performing a second autonomous function associated with the surgical task. The second autonomous function may be associated with fault mitigation feedback.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to the following concurrently filed patent applications, the contents of each of which are incorporated herein by reference:

[0003] U.S. Patent Application Titled “METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM,” Attorney Docket No. END9430USNP1.

[0004] U.S. Patent Application Titled “DYNAMICALLY DETERMINING SURGICAL AUTONOMY LEVEL,” Attorney Docket No. END9430USNP2. Background Art

[0005] Surgical procedures are typically performed in a surgical suite or operating room at a medical facility, such as, for example, a hospital. Various surgical devices and systems are utilized in performing surgical procedures. In the digital and information age, medical systems and facilities are often slow to implement systems or procedures that utilize newer and improved technologies due to patient safety and a general desire to maintain traditional practices. Summary of the invention

[0006] This article describes systems, methods, and instruments for detecting fault mitigation associated with a surgical task. A device may perform a first autonomous function associated with a surgical task. Performing the first autonomous function may be associated with a control feedback. The first autonomous function may be monitored by tracking one or more outputs associated with the control feedback. In the event that the one or more outputs exceed a fault threshold, the device may switch from performing the first autonomous function to performing a second autonomous function associated with the surgical task. The second autonomous function may be associated with a fault mitigation feedback. The device may generate a fault magnitude based on the one or more outputs that exceed the fault threshold. The device may adjust one or more parameters associated with the fault mitigation feedback based on the fault magnitude. In an example, the fault magnitude may be generated based on one or more of the following: a surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or a degree to which the one or more outputs exceed the fault threshold.

[0007] A device for controlling the performance of a surgical task by a surgical instrument is described. The device includes a processor. The processor is configured to monitor a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task. The processor is configured to control the surgical instrument according to a first model based on the tracked one or more outputs. The processor is configured to switch the surgical instrument from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model in the event that at least one of the tracked one or more outputs exceeds a fault threshold. Advantageously, these features enable the combination of the device and the surgical instrument to act as a self-contained and fault-resistant autonomous surgical system. Many instrument controllers that use any kind of output or feedback to control instruments will provide control instructions that are a direct function of the tracked output (e.g., intended to keep the output constant, set a parameter to a linear function of the output value, etc.). Therefore, if a very abnormal or unexpected output is detected (usually associated with some kind of fault), the control instructions provided to the instrument based on the output may also be abnormal and may be inappropriate (or even unsafe). In the present invention this is avoided because the control model is changed when the tracked output exceeds the fault threshold.The device may be external to the surgical instrument, but may also be a processor or control circuit contained within the surgical instrument.

[0008] The first autonomous function and the second autonomous function may be different autonomous functions. This may be beneficial because it may allow the function to be changed when a fault is detected so that the procedure can be safely paused, stopped, interrupted or completed. For example, the first autonomous function may include a clamping operation and if a threshold force is exceeded, the second autonomous function may be a jaw opening operation.

[0009] Controlling the surgical instrument according to the first model may include providing control feedback generated by the first model to the surgical instrument. The control feedback may be associated with setting and / or updating one or more parameters of the surgical instrument. The one or more parameters of the surgical instrument may include parameters of a script for controlling the surgical instrument.

[0010] The control feedback may be configured to be usable as an input to one or more actuators of the surgical instrument.The control feedback may be configured to be capable of setting parameters via a message to the surgical instrument.

[0011] One or more outputs associated with the first autonomous function and / or the surgical task may include measured real-world data derived from one or more of: measurements from sensors on the surgical instrument; measurements from wearable sensors, optionally including heart rate, that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task; and measurements of the position of one or more actuators on the surgical instrument performing the surgical task.

[0012] Controlling the surgical instrument according to the first model based on the tracked outputs may include any of: controlling the surgical instrument according to at least one output; or comparing at least one output with an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and controlling the surgical instrument according to the difference.

[0013] Determining whether at least one of the one or more outputs has exceeded a failure threshold may include any of the following: determining that the at least one output may include measured real-world data that falls outside a given range; or comparing the at least one output may include comparing the measured real-world data with an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and determining that the difference falls outside a given range.

[0014] Determining whether at least one of the one or more outputs has exceeded a fault threshold may include determining whether each of a predetermined number of the one or more outputs has exceeded the fault threshold. The predetermined number may be manually input by a user of the surgical instrument.

[0015] Controlling the surgical instrument according to the fault mitigation model may include one or more of the following: generating or setting one or more parameters for the surgical instrument that are known or expected to reduce the risk associated with the completion of the surgical task; sending a message to the user of the surgical instrument; reducing the autonomy level of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting; terminating all operations related to the surgical task; or reducing system performance to enable safe completion of the surgical task. Switching the surgical instrument from performing the first autonomous function to performing the second autonomous function associated with the surgical task may correspond to switching to a less autonomous control, switching an operating parameter of the device (e.g., reducing the speed of the cutter, reversing the cutter, etc.), switching to an algorithm that stops operation (e.g., switching from clamping to opening), or changing the power level of an energy device, etc. Advantageously, each of these fault mitigation responses may provide a response to a detected fault during the performance of the surgical task, which may reduce the risk of harm to the patient. For example, terminating all operations related to the surgical task may immediately stop the erroneous autonomous function, thereby preventing the erroneous autonomous function from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation measures described above that do not simply terminate operation may provide benefits over the original fault mitigation control that simply stops the process immediately. This is because, in various surgical contexts, better clinical outcomes (e.g., reduced harm to the patient or reduced complications) may be achieved by continuing to operate the instrument in a safety-oriented manner rather than terminating operation of the instrument. For example, a surgical instrument having an end effector for clamping tissue, whose clamping arms are biased into a closed position, may be made safe by a controller by setting an actuator to release the jaws of the end effector and thereby release any tissue held therein.

[0016] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamping jaws of the end effector. During "normal" operation (i.e., operation according to the first model), the clamping force feedback may be used to control the speed of an actuator that is configured to be able to advance a knife member and / or a slider assembly. This may include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model may indicate the adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a large force is detected (indicating a larger and more robust tissue type), and slightly reducing the firing speed when a small force is detected (indicating a smaller and more fragile tissue type). However, if it is determined that the force on the jaws has dropped below a fault threshold (e.g., has completely dropped to 0 or almost dropped to 0), this may indicate that the tissue has slipped out of the jaws of the end effector. Therefore, the system may switch to a fault mitigation model that may indicate setting the firing speed to a negative value (corresponding to reversing the firing direction and driving the knife back into its recessed housing) because this is a known (or at least reasonably expected) way to ensure the safety of the instrument by allowing the end effector to be repositioned without a serious risk of injury to the patient or operator. The system may also optionally alert the user to the fact that this is occurring.

[0017] As another non-limiting example, when an unexpectedly high force is detected closing the end effector of a surgical instrument, the system may switch to a fault mitigation mode that may gradually increase or decrease the motor speed of the surgical instrument.

[0018] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply radio frequency (RF) and ultrasonic energy to tissue may be controlled according to a first model while monitoring the impedance level at the end effector electrode. The first model may cause RF energy to be applied to the tissue and may adjust energy delivery parameters (e.g., of a generator) based on the monitored impedance to ensure that energy delivery (e.g., power) at the surgical site remains substantially constant. However, if an unexpected peak in impedance (very high or very low) is detected, this may exceed a fault threshold (which is defined as an impedance range within which the detected impedance is expected to fall during normal operation of the device). Therefore, a switch may be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example may include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent undesired tissue transection.

[0019] The tracked one or more outputs may be the tracked first one or more outputs. The processor may be further configured to monitor the second autonomous function associated with the surgical task by tracking the second one or more outputs associated with the second autonomous function and / or the surgical task. The processor may be further configured to control the surgical instrument according to the fault mitigation model may be based on the tracked second one or more outputs.

[0020] The processor may be further configured to generate a fault threshold based on the one or more outputs if the one or more outputs exceed the fault threshold. Controlling the surgical instrument according to the fault mitigation model may include adjusting one or more parameters of the surgical instrument based on the magnitude of the fault. Advantageously, this intelligently adjusts the response of the device and instrument to the fault in proportion to the magnitude of the fault.

[0021] The fault magnitude may be generated based on at least one of the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold. Advantageously, this enables detection of faults that may be critical but not numerous (or vice versa), for example, thereby allowing the device to strike a balance between importance and quantity when mitigating faults. Additionally or alternatively, this advantageously allows more important surgical functions (e.g., life-threatening) to receive fault mitigation in preference to less important surgical functions (e.g., practicality / surgeon ease of use).

[0022] The processor may be further configured to: perform an analysis on the tracked one or more outputs. The processor may be further configured to: generate a comparison output based on the analysis. The processor may be further configured to: in the event that the comparison output exceeds the fault threshold, perform the switch from performing the first autonomous function to performing the second autonomous function. Advantageously, in this way, the switching of autonomous functions may be performed in response to relative output problems rather than absolute output problems (e.g., which may be compared to an ideal, expected and / or conceptual level of output). Additionally or alternatively, it may be beneficial to perform such function switching in response to a relationship between one or more outputs, thereby allowing for more nuance.

[0023] The processor may be further configured to be capable of: assigning a weight to each of the one or more tracked outputs based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the extent to which the one or more outputs exceed the fault threshold.

[0024] The processor may be further configured to: send an indication of a fault message to a user of a surgical instrument associated with the surgical task if the one or more outputs exceed the fault threshold. The fault message may include a fault type and a fault magnitude.

[0025] The fault message may include a set of recommendations based on: the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0026] The processor may be further configured to determine the fault threshold and the fault mitigation feedback based on training data.

[0027] The training data may be based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0028] This failure threshold may be adjusted based on situational awareness.

[0029] A computer-implemented method for controlling the performance of a surgical task by a surgical instrument is described. The method includes monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task. The method includes providing a digital command or user recommendation for controlling the surgical instrument according to a first model based on the tracked one or more outputs. The method also includes providing a digital command or user recommendation for switching the surgical instrument from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model based on determining that at least one of the one or more outputs has exceeded a fault threshold. Advantageously, these features enable the combination of a computer implementing the method and the surgical instrument to act as a self-contained and fault-resistant autonomous surgical system. Many instrument control methods that use any kind of output or feedback to control an instrument involve providing control instructions that are a direct function of the tracked output (e.g., intended to keep the output constant, set a parameter to a linear function of the output value, etc.). Therefore, if a very abnormal or unexpected output (usually associated with some kind of fault) is detected, the control instructions provided to the instrument based on the output may also be abnormal and may be inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds the fault threshold.The computer implementing the method may be external to the surgical instrument, but may also be a processor or control circuit contained within the surgical instrument itself.

[0030] The first autonomous function and the second autonomous function may be different autonomous functions. This may be beneficial because it may allow the function to be changed when a fault is detected so that the procedure can be safely paused, stopped, interrupted or completed. For example, the first autonomous function may include a clamping operation and if a threshold force is exceeded, the second autonomous function may be a jaw opening operation.

[0031] The digital command or the user recommendation for controlling the surgical instrument according to the first model may include one or more values ​​for setting or updating parameters of the surgical instrument. The one or more parameters of the surgical instrument may include parameters of a script for controlling the surgical instrument.

[0032] These values ​​may be suitable for use as input to one or more actuators of the surgical instrument.The digital command may be a message to the surgical instrument configured to enable setting of a parameter of the surgical instrument.

[0033] The one or more outputs associated with the first autonomous function and / or the surgical task may include measured real-world data derived from one or more of: measurements of sensors on the surgical instrument. The outputs may also include measurements of wearable sensors, optionally including heart rate, that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task. The outputs may include measurements of the position of one or more actuators on the surgical instrument performing the surgical task.

[0034] The digital command or the user recommendation for controlling the surgical instrument according to the first model based on the tracked output can be any of the following: a digital command or user recommendation for controlling the surgical instrument according to at least one output; or a digital command or user recommendation for controlling the surgical instrument according to a difference determined by comparing at least one output with an ideal output generated by the first model.

[0035] Determining whether at least one of the one or more outputs has exceeded a failure threshold may include any of the following: determining that at least one output includes measured real-world data that falls outside a given range; or comparing at least one output including measured real-world data with an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and determining that the difference falls outside a given range.

[0036] Determining whether at least one of the one or more outputs has exceeded a fault threshold may include determining whether each of a predetermined number of the one or more outputs has exceeded the fault threshold. The predetermined number may be manually input by a user of the surgical instrument.

[0037] The digital command or user recommendation for switching the surgical instrument from performing the first autonomous function to performing the second autonomous function may include a digital command or user recommendation for the following operations: generating or setting one or more parameters for the surgical instrument that are known or expected to reduce the risk associated with the completion of the surgical task; sending a message to the user of the surgical instrument; reducing the autonomy level of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting; terminating all operations related to the surgical task; or reducing system performance to enable safe completion of the surgical task. Switching the surgical instrument from performing the first autonomous function to performing the second autonomous function associated with the surgical task may correspond to switching to a less autonomous control, switching an operating parameter of the device (e.g., reducing the speed of the cutter, reversing the cutter, etc.), switching to an algorithm that stops operation (e.g., switching from clamping to opening), or changing the power level of an energy device, etc. Advantageously, each of these fault mitigation responses can provide a response to a detected fault during the performance of the surgical task, which can reduce the risk of harm to the patient. For example, terminating all operations related to the surgical task may immediately stop the erroneous autonomous function, thereby preventing the erroneous autonomous function from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation measures described above (which do not simply terminate the operation) may provide benefits over the original fault mitigation control that simply stops the process immediately. This is because, in various surgical contexts, better clinical results (e.g., reduced harm to the patient or reduced complications) may be achieved by continuing to operate the instrument in a safety-oriented manner compared to terminating the operation of the instrument. For example, a surgical instrument having an end effector for clamping tissue (whose clamping arms are biased into a closed position) may be ensured safe by the controller by setting the actuator to release the jaws of the end effector and thereby release any tissue held therein.

[0038] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamping jaws of the end effector. During "normal" operation (i.e., operation according to the first model), the clamping force feedback may be used to control the speed of an actuator that is configured to be able to advance a knife member and / or a slider assembly. This may include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model may indicate the adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a large force is detected (indicating a larger and more robust tissue type), and slightly reducing the firing speed when a small force is detected (indicating a smaller and more fragile tissue type). However, if it is determined that the force on the jaws has dropped below a fault threshold (e.g., has completely dropped to 0 or almost dropped to 0), this may indicate that the tissue has slipped out of the jaws of the end effector. Therefore, the system may switch to a fault mitigation model that may indicate setting the firing speed to a negative value (corresponding to reversing the firing direction and driving the knife back into its recessed housing) because this is a known (or at least reasonably expected) way to ensure the safety of the instrument by allowing the end effector to be repositioned without a serious risk of injury to the patient or operator. The system may also optionally alert the user to the fact that this is occurring.

[0039] As another non-limiting example, when an unexpectedly high force is detected closing the end effector of a surgical instrument, the system may switch to a fault mitigation mode that may gradually increase or decrease the motor speed of the surgical instrument.

[0040] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply radio frequency (RF) and ultrasonic energy to tissue may be controlled according to a first model while monitoring the impedance level at the end effector electrode. The first model may cause RF energy to be applied to the tissue and may adjust energy delivery parameters (e.g., of a generator) based on the monitored impedance to ensure that energy delivery (e.g., power) at the surgical site remains substantially constant. However, if an unexpected peak in impedance (very high or very low) is detected, this may exceed a fault threshold (which is defined as an impedance range within which the detected impedance is expected to fall during normal operation of the device). Therefore, a switch may be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example may include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent undesired tissue transection.

[0041] The tracked one or more outputs may be the tracked first one or more outputs. The method may also include monitoring the second autonomous function associated with the surgical task by tracking the second one or more outputs associated with the second autonomous function and / or the surgical task. The digital command or the user recommendation for controlling the surgical instrument according to the fault mitigation model may be based on the tracked second one or more outputs.

[0042] The method may also include: generating a fault threshold based on the one or more outputs if the one or more outputs exceed the fault threshold. The digital command or the user recommendation may be to adjust one or more parameters of the surgical instrument based on the magnitude of the fault. Advantageously, this intelligently adjusts the computer and instrument response to the fault in proportion to the magnitude of the fault.

[0043] The fault magnitude may be generated based on at least one of the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold. Advantageously, this enables detection of faults that may be critical but not numerous (or vice versa), for example, thereby allowing the computer to strike a balance between importance and quantity when mitigating faults. Additionally or alternatively, this advantageously allows more important surgical functions (e.g., life-threatening) to receive fault mitigation in preference to less important surgical functions (e.g., practicality / surgeon ease of use).

[0044] The method may further include performing an analysis on the tracked one or more outputs. The method may further include generating a comparison output based on the analysis. Based on determining that the comparison output has exceeded the fault threshold, the digital command or the user recommendation for switching the surgical instrument from performing the first autonomous function to performing the second autonomous function may be provided. Advantageously, in this manner, switching of autonomous functions may be performed in response to relative output issues rather than absolute output issues (e.g., comparable to an ideal, expected, and / or conceptual level of output). Additionally or alternatively, it may be beneficial to perform such a function switch in response to a relationship between one or more outputs, thereby allowing for more nuance.

[0045] The method may also include assigning a weight to each of the one or more tracked outputs based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the extent to which the one or more outputs exceed the fault threshold.

[0046] The method may also include sending an indication of a fault message to a user of a surgical instrument associated with the surgical task if the one or more outputs exceed the fault threshold. The fault message may include a fault type and a fault magnitude.

[0047] The fault message may include a set of recommendations based on: the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0048] The method may further include determining the fault threshold and fault mitigation feedback based on training data.

[0049] The training data may be based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0050] This fault threshold may be adjusted based on situational awareness.

[0051] A computer program product is described for causing a computer to perform any of the previously described methods.

[0052] A non-transitory computer-readable storage medium is described that includes computer-readable instructions that, when executed by a computer, cause the computer to perform a method in any of the previously described methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a block diagram of a computer-implemented surgical system.

[0054] Figure 2 An exemplary surgical system is shown in a surgical operating room.

[0055] Figure 3 Exemplary surgical hubs that pair with various systems are illustrated.

[0056] Figure 4 A surgical data network having a set of communicating surgical hubs configured to be able to connect to a set of sensing systems, an environmental sensing system, a set of devices, etc. is illustrated.

[0057] Figure 5 A logic diagram of a control system for a surgical instrument is illustrated.

[0058] Figure 6An exemplary surgical system is shown that includes a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter.

[0059] Figure 7 An exemplary situational awareness surgical system is shown.

[0060] Figure 8 An example of a surgical autonomous system is shown.

[0061] Fig. 9 The relationship between control feedback and fault mitigation feedback is shown.

[0062] Fig.10 An example of performing autonomous functions using ideal models and fault models is shown.

[0063] Fig.11 An example of utilizing control feedback and fault mitigation feedback to perform autonomous functions is shown.

[0064] Fig.12 An example of an overview of a surgical instrument and a surgical hub with control feedback and fault mitigation feedback is shown. DETAILED DESCRIPTION

[0065] Figure 1 20000. An exemplary surgical system such as surgical system 20000 may include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, surgical system 20002 may include a computer-implemented interactive surgical system. For example, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 that communicates with a cloud computing system 20008, such as Figure 2 The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Exemplary surgical systems 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human-machine interface system 20012, and the like. The human-machine interface system is also referred to herein as a human-machine interface device. The wearable sensing system 20011 may include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include, for example, one or more devices for measuring one or more environmental attributes, such as Figure 2 As further described. The robotic system 20013 may include a plurality of devices for performing a surgical procedure, such as Figure 2 Further described.

[0066] The surgical system 20002 can communicate with a remote server 20009, which can be part of a cloud computing system 20008. In one example, the surgical system 20002 can communicate with the remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, the patient sensing system can communicate directly with the remote server 20009. The surgical system 20002 and / or components therein can communicate with the remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, Advanced LTE (LTE-A), New Radio (NR), or 5G.

[0067] The surgical hub 20006 can cooperate and interact with one of the multiple devices that display images from the laparoscope and information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from one or more sensing systems 20011 and send notification or control messages to the one or more sensing systems 20011. The surgical hub 20006 can send information including notification information to the human-machine interface system 20012 and / or receive information including notification information from the human-machine interface system. The human-machine interface system 20012 may include one or more human-machine interface devices (HIDs). The surgical hub 20006 can send and / or receive notification information or control information to convert into audio, display and / or control information to various devices communicating with the surgical hub.

[0068] For example, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, such as Figure 1 The one or more sensing systems 20001 can measure data related to various biomarkers. The one or more sensing systems 20001 can use one or more sensors such as light sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, pyroelectric sensors, infrared sensors, etc. to measure biomarkers. The one or more sensors can use one or more of the following sensing technologies to measure biomarkers as described herein: photoplethysmography, electrocardiography, electroencephalography, colorimetry, impedance tracing, potentiometric measurement, amperometric measurement, etc.

[0069] Biomarkers measured by the one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximum oxygen uptake, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood pH, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, coughing and sneezing, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, psychiatric factors, sweat, circulating tumor cells, autonomic nervous tone, circadian rhythm and / or menstrual cycle.

[0070] Biomarkers may be related to physiological systems, which may include, but are not limited to, behavior and psychology, cardiovascular system, renal system, skin system, nervous system, gastrointestinal system, respiratory system, endocrine system, immune system, tumor, musculoskeletal system and / or reproductive system. Information from biomarkers can be determined and / or used by, for example, a computer-implemented patient and surgical system 20000. Information from biomarkers can be determined and / or used by a computer-implemented patient and surgical system 20000, for example, to improve the system and / or improve patient outcomes. One or more sensing systems 20001, biomarkers 20005 and physiological systems are described in more detail in U.S. application 17 / 156,287 (Agent File No. END9290USNP1), filed on January 22, 2021, entitled "METHODOF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS", the disclosure of which is incorporated herein by reference in its entirety.

[0071] Figure 2 An example of a surgical system 20002 in a surgical operating room is shown. Figure 2 As shown, a patient undergoes surgery by one or more healthcare professionals (HCPs). The HCP is monitored by one or more HCP sensing systems 20020 worn by the HCP. The HCP and the environment around the HCP may also be monitored by one or more environmental sensing systems, which may include, for example, a set of cameras 20021, a set of microphones 20022, and other sensors that may be deployed in an operating room. The HCP sensing system 20020 and the environmental sensing system may communicate with a surgical hub 20006, which in turn may communicate with one or more cloud servers 20009 of a cloud computing system 20008, such as Figure 1 The environmental sensing system may be used to measure one or more environmental attributes, such as the position of the HCP in the operating room, HCP movement, ambient noise in the operating room, temperature / humidity in the operating room, etc.

[0072] like Figure 2As shown, the main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned in the sterile area to be visible to the operator at the operating table 20024. In addition, the visualization / notification tower 20026 is positioned outside the sterile area. The visualization / notification tower 20026 may include a first non-sterile human-machine interaction device (HID) 20027 and a second non-sterile HID 20029 that are separated from each other. The HID can be a display or a display with a touch screen that allows people to directly interface with the HID. The human-machine interface system guided by the surgical hub 20006 can be configured to be able to coordinate the information flow of operators inside and outside the sterile area using HIDs 20027, 20029, and 20023. In one example, the surgical hub 20006 can cause the HID (e.g., the main HID 20023) to display notifications and / or information about patients and / or surgical procedures. In one example, the surgical hub 20006 can prompt for input and / or receive input from personnel in the sterile or non-sterile field. In one example, the surgical hub 20006 can cause the HID to display a snapshot of the surgical site recorded by the imaging device 20030 on the non-sterile HID 20027 or 20029 while maintaining a real-time feed of the surgical site on the main HID 20023. For example, the snapshot on the non-sterile display 20027 or 20029 can allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0073] In one aspect, the surgical hub 20006 can be configured to be able to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to the primary display 20023 within the sterile field, where the sterile operator at the operating table can view the diagnostic input or feedback. In one example, the input can be a modified form of a snapshot displayed on the non-sterile display 20027 or 20029, which can be routed to the primary display 20023 via the surgical hub 20006.

[0074] refer to Figure 2, surgical instrument 20031 is used in a surgical procedure as part of surgical system 20002. Hub 20006 can be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, described in U.S. Patent Application Publication No. US2019-0200844A1 (U.S. Patent Application No. 16 / 209,385) filed on December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at visualization tower 20026 can be routed by hub 20006 to a surgical instrument display within the sterile field, where the operator of surgical instrument 20031 can view the diagnostic input or feedback. For example, exemplary surgical instruments suitable for use with surgical system 20002 are described under the heading “Surgical Instrument Hardware” in U.S. patent application publication No. US 2019-0200844A1 (U.S. patent application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0075] Figure 2 An example of a surgical system 20002 for performing a surgical procedure on a patient lying flat on an operating table 20024 in a surgical operating room 20035 is shown. A robotic system 20034 may be used in a surgical procedure as part of the surgical system 20002. The robotic system 20034 may include a surgeon's console 20036, a patient side cart 20032 (surgical robot), and a surgical robotic hub 20033. The patient side cart 20032 may manipulate at least one removably coupled surgical tool 20037 through a minimally invasive incision in the patient's body while the surgeon observes the surgical site through the surgeon's console 20036. Images of the surgical site may be obtained through a medical imaging device 20030, which may be manipulated by the patient side cart 20032 to orient the imaging device 20030. The robotic hub 20033 may be used to process images of the surgical site for subsequent display to the surgeon through the surgeon's console 20036.

[0076] Other types of robotic systems may be readily adapted for use with the surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application No. US2019-0201137A1, filed on December 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL” (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0077] Various examples of cloud-based analytics performed by cloud computing system 20008 and suitable for use with the present disclosure are described in U.S. patent application publication No. US2019-0206569 A1 (U.S. patent application No. 16 / 209,403), entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0078] In various aspects, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0079] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0080] One or more illumination sources may be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescent spectrum) is that portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye) and may be referred to as visible light or simply light. A typical human eye will respond to wavelengths in the range of about 380 nm to about 750 nm in air.

[0081] The invisible spectrum (e.g., non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0082] In various aspects, the imaging device 20030 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophago-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0083] The imaging device can use multispectral monitoring to distinguish the topography and the underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. US2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385) filed on December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY", the disclosure of which is incorporated herein by reference in its entirety. After completing the surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for relocating the surgical site. It is self-evident that during any surgical operation, the operating room and surgical equipment need to be strictly sterilized. The strict sanitation and sterilization conditions required in the "surgical room" (i.e., operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including the imaging device 20030 and its attachments and components. It should be understood that the sterile field can be considered as a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered as the area around the patient prepared for surgery. The sterile field can include properly dressed scrubbed team members, as well as all equipment and fixtures in the area.

[0084] Figure 1 The wearable sensing system 20011 shown may include one or more sensing systems, such as Figure 2HCP sensing system 20020 shown. HCP sensing system 20020 may include a sensing system for monitoring and detecting a set of physical states and / or a set of physiological states of a healthcare provider (HCP). The HCP may typically be a surgeon or one or more healthcare providers assisting the surgeon or other healthcare providers. In one example, sensing system 20020 may measure a set of biomarkers to monitor the heart rate of the HCP. In one example, a sensing system 20020 (e.g., a watch or wristband) worn on the surgeon's wrist may use an accelerometer to detect hand motion and / or shaking and determine the amplitude and frequency of the tremor. Sensing system 20020 may send measurement data associated with the set of biomarkers and data associated with the surgeon's physical state to surgical hub 20006 for further processing. One or more environmental sensing devices may send environmental information to surgical hub 20006. For example, an environmental sensing device may include a camera 20021 for detecting the hand / body position of the HCP. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include devices such as thermometers for measuring temperature and hygrometers for measuring humidity of the environment in the operating room. A surgical hub 20006, either alone or in communication with a cloud computing system, may use surgeon biomarker measurement data and / or environmental sensing information to modify the average latency of a control algorithm or a robotic interface of a handheld instrument, for example, to minimize tremor. In one example, the HCP sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and send measurement data associated with the surgeon biomarkers to the surgical hub 20006. The HCP sensing system 20020 may communicate with the surgical hub 20006 using one or more of the following RF protocols: Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Wi-Fi. Surgeon biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels associated with surgeons or patients, surgeon and / or personnel movement, surgeon and / or personnel attention levels, etc.

[0085] The surgical hub 20006 can use the surgeon biomarker measurement data associated with the HCP to adaptively control one or more surgical instruments 20031. For example, the surgical hub 20006 can send a control program to the surgical instrument 20031 to control its actuator to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 can send the control program based on situational awareness and / or context about the importance or criticality of the task. When control is needed, the control program can instruct the instrument to change operation to provide more control.

[0086] Figure 3 An exemplary surgical system 20002 with a surgical hub 20006 is shown. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human-machine interface system 20012, a robotic system 20013, and a smart device 20014 via modular controls. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain aspects, as Figure 3 As shown, the hub 20006 also includes a smoke evacuation module 20054 and / or a suction / irrigation module 20055. Various modules and systems can be connected to the modular controls directly via a router or via a communication module 20056. The operating room devices can be coupled to cloud computing resources and data storage devices via the modular controls. The human-machine interface system 20012 may include a display subsystem and a notification subsystem.

[0087] The modular control may be coupled to a non-contact sensor module. The non-contact sensor module may use an ultrasonic, laser-type and / or similar non-contact measuring device to measure the size of the operating room and generate a mapping of the surgical room. Other distance sensors may be used to determine the boundaries of the operating room. The ultrasonic-based non-contact sensor module may scan the operating room by emitting a burst of ultrasound and receiving an echo when it bounces back from the walls of the operating room, as described in the title "Surgical Hub Spatial Awareness Within an Operating Room" in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled "INTERACTIVE SURGICAL PLATFORM" filed on December 28, 2017, which is incorporated herein by reference in its entirety. The sensor module may be configured to be able to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module may scan the operating room by emitting a laser pulse, receiving a laser pulse that bounces back from the walls of the operating room, and comparing the phase of the emitted pulse with the received pulse to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0088] During surgery, the application of energy to tissue for sealing and / or cutting is often associated with smoke evacuation, suction of excess fluid, and / or flushing of tissue. Fluid lines, power lines, and / or data lines from different sources are often tangled during surgery. Solving this problem during surgery can waste valuable time. Disconnecting a line may require disconnecting the line from its corresponding module, which may require resetting the module. The hub modular housing 20060 provides a unified environment for managing power lines, data lines, and fluid lines, which reduces the frequency of tangling between such lines. Various aspects of the present disclosure provide a surgical hub 20006 for use in surgery involving application of energy to tissue at a surgical site. The surgical hub 20006 includes a hub housing 20060 and a combined generator module slidably accommodated in a docking base of the hub housing 20060. The docking base includes data contacts and power contacts. The combined generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component seated in a single unit. In one aspect, the combined generator module also includes a smoke exhaust component for connecting the combined generator module to at least one energy delivery cable of a surgical instrument, at least one smoke exhaust component configured to be able to exhaust smoke, fluid and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke exhaust component. In one aspect, the fluid line can be a first fluid line, and the second fluid line can extend from the remote surgical site to the suction and flushing module 20055 slidably accommodated in the hub housing 20060. In one aspect, the hub housing 20060 may include a fluid interface. Certain surgical procedures may require more than one energy type to be applied to the tissue. One energy type may be more conducive to cutting tissue, while another different energy type may be more conducive to sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of the present disclosure provide a solution in which a hub modular housing 20060 is configured to accommodate different generators and facilitate interactive communication between them. One of the advantages of the hub modular housing 20060 is that it enables rapid removal and / or replacement of various modules. Aspects of the present disclosure provide a modular surgical housing for use in surgical procedures involving application of energy to tissue. The modular surgical housing includes a first energy generator module, the first energy generator module is configured to generate a first energy for application to tissue, and a first docking base, the first docking base includes a first docking port, the first docking port includes a first data and power contact, wherein the first energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the first energy generator module is slidably movable out of electrical engagement with the first power and data contacts.Further describing the above, the modular surgical shell also includes a second energy generator module, the second energy generator module is configured to generate a second energy different from the first energy for application to tissue, and a second docking base, the second docking base includes a second docking port, the second docking port includes second data and power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the second energy generator is slidably movable out of electrical contact with the second power and data contacts. In addition, the modular surgical shell also includes a communication bus between the first docking port and the second docking port, which is configured to facilitate communication between the first energy generator module and the second energy generator module. Reference. Figure 3 , aspects of the present disclosure are presented as a hub modular housing 20060 that allows modular integration of a generator module 20050, a smoke evacuation module 20054, and a suction / irrigation module 20055. The hub modular housing 20060 also facilitates interactive communication between the modules 20059, 20054, 20055. The generator module 20050 may have integrated monopolar components, bipolar components, and ultrasonic components supported in a single housing unit that is slidably inserted into the hub modular housing 20060. The generator module 20050 may be configured to be connectable to a monopolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 20060. The hub modular housing 20060 can be configured to facilitate the insertion of multiple generators and interactive communication between the generators docked into the hub modular housing 20060 so that the generators will function as a single generator.

[0089] Figure 4 A surgical data network having a set of communication hubs according to at least one aspect of the present disclosure is shown, the communication hubs being configured to connect a set of sensing systems, environmental sensing systems, and a set of other modular devices located in one or more operating rooms, patient recovery rooms, or rooms in the medical facility specifically equipped for surgical operations to a cloud.

[0090] like Figure 4As shown, the surgical hub system 20060 may include a modular communication hub 20065 that is configured to connect modular devices located in a medical facility to a cloud-based system (e.g., a cloud computing system 20064, which may include a remote server 20067 coupled to a remote storage device 20068). The modular communication hub 20065 and the devices may be connected in a room in the medical facility that is specifically equipped for surgical operations. In one aspect, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicates with a network router 20066. The modular communication hub 20065 may be coupled to a local computer system 20063 to provide local computer processing and data manipulation.

[0091] The computer system 20063 may include a processor and a network interface 20100. The processor may be coupled to a communication module, a storage device, a memory, a non-volatile memory, and an input / output (I / O) interface via a system bus. The system bus may be any of several types of bus structures including a memory bus or a memory controller, a peripheral bus or an external bus, and / or a local bus using any of a variety of available bus architectures, including but not limited to a 9-bit bus, an Industry Standard Architecture (ISA), a Micro Charmel Architecture (MSA), an Extended ISA (EISA), Intelligent Drive Electronics (IDE), a VESA Local Bus (VLB), a Peripheral Component Interconnect (PCI), a USB, an Advanced Graphics Port (AGP), a Personal Computer Memory Card International Association bus (PCMCIA), a Small Computer System Interface (SCSI), or any other peripheral bus.

[0092] The controller may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be a commercially available LM4F230H5QR ARM Cortex-M4F processor core, such as Texas Instruments, which includes 256KB of single-cycle flash or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improved execution above 40MHz, 32KB of single-cycle sequential random access memory (SRAM), a 32KB MCU loaded with Internal read-only memory (ROM) for software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which can be found in the product data sheet.

[0093] In one example, the processor may include a safety controller including two families based on controllers such as TMS570 and RM4x, also known by Texas Instruments under the trade name Hercules ARMCortex R4. The safety controller may be configured to be capable of being specifically used for IEC 61508 and ISO 26262 safety-critical applications, among others, to provide advanced integrated safety features while delivering scalable execution, connectivity, and memory options.

[0094] It should be understood that the computer system 20063 may include software that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment. Such software may include an operating system. An operating system that may be stored on a disk storage device may be used to control and allocate the resources of a computer system. System applications may utilize the operating system to manage resources through program modules and program data stored in system memory or disk storage devices. It should be understood that the various components described herein may be implemented using various operating systems or combinations of operating systems.

[0095] The user can input commands or information into the computer system 20063 through an input device coupled to the I / O interface. The input device may include, but is not limited to, a pointing device, such as a mouse, a touch ball, a stylus, a touch pad, a keyboard, a microphone, a joystick, a game pad, a satellite dish, a scanner, a TV tuner card, a digital camera, a digital video camera, a web camera, etc. These and other input devices are connected to the processor 20102 through the system bus via an interface port. The interface port includes, for example, a serial port, a parallel port, a game port, and a USB. The output device uses the same type of port as the input device. Therefore, for example, a USB port can be used to provide input to the computer system 20063 and output information from the computer system 20063 to the output device. An output adapter is provided to illustrate that some output devices, such as monitors, displays, speakers, and printers, may need special adapters in other output devices. The output adapter may include, by way of example, but is not limited to, a video and sound card that provides a connection device between the output device and the system bus. It should be noted that other devices or systems such as remote computers can provide both input capabilities and output capabilities.

[0096] Computer system 20063 can be operated in a networked environment using a logical connection with one or more remote computers (such as cloud computers) or local computers. A remote cloud computer can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device or other public network node, etc., and generally includes many or all of the elements described relative to the computer system. For simplicity, only a memory storage device with a remote computer is shown. A remote computer can be logically connected to a computer system via a network interface, and then physically connected via a communication connection. A network interface can cover communication networks such as a local area network (LAN) and a wide area network (WAN). LAN technology can include a fiber distributed data interface (FDDI), a copper distributed data interface (CDDI), Ethernet / IEEE 802.3, a token ring / IEEE 802.5, etc. WAN technology can include, but is not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and variants thereof, packet switching networks, and digital subscriber lines (DSL).

[0097] In various examples, computer system 20063 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) for processing digital images. The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instructions, multiple data (MIMD) techniques to improve speed and efficiency. The digital image processing engine may perform a range of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0098] The communication connection may refer to the hardware / software used to connect the network interface to the bus. Although the communication connection is shown for exemplary clarification within the computer system 20063, it may also be located outside the computer system 20063. The hardware / software necessary to connect to the network interface may include internal and external technologies for exemplary purposes only, such as modems, including conventional telephone-grade modems, cable modems, fiber optic modems and DSL modems, ISDN adapters, and Ethernet cards. In some examples, an RF interface may also be used to provide a network interface.

[0099] The surgical data network associated with the surgical hub system 20060 can be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, enabling it to be transferred from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network includes additional features to enable monitoring of traffic through the surgical data network and configuration of each port in the network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0100] Modular devices 1a-1n located in an operating room may be coupled to a modular communication hub 20065. A network hub 20061 and / or a network switch 20062 may be coupled to a network router 20066 to connect devices 1a-1n to a cloud computing system 20064 or a local computer system 20063. Data associated with devices 1a-1n may be transmitted to a cloud-based computer via a router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transmitted to a local computer system 20063 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be coupled to a network switch 20062. A network switch 20062 may be coupled to a network hub 20061 and / or a network router 20066 to connect devices 2a-2m to a cloud 20064. Data associated with devices 2a-2m may be transmitted to a cloud computing system 20064 via a network router 20066 for data processing and manipulation. Data associated with the devices 2a-2m may also be transmitted to the local computer system 20063 for local data processing and manipulation.

[0101] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing system 20069 may include an HCP sensing system and / or a patient sensing system. The one or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly via one of the network routers 20066 or via a network hub 20061 or a network switch 20062 that communicates with the network router 20066.

[0102] The sensing system 20069 may be coupled to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 may be transmitted to the cloud computing system 20064 for data processing and manipulation via the network router 20066. Data associated with the sensing system 20069 may also be transmitted to the local computer system 20063 for local data processing and manipulation.

[0103] like Figure 4 As shown, the surgical hub system 20060 can be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. A modular communication hub 20065 can be included in a modular control tower that is configured to accommodate multiple devices 1a-1n / 2a-2m. A local computer system 20063 can also be included in the modular control tower. The modular communication hub 20065 can be connected to a display 20068 to display images obtained by some of the devices 1a-1n / 2a-2m during a surgical procedure, for example. In various aspects, the devices 1a-1n / 2a-2m can include, for example, various modules, such as an imaging module coupled to an endoscope, a generator module coupled to an energy-based surgical device, a smoke evacuation module, a suction / irrigation module, a communication module, a processor module, a storage array, a surgical device connected to a display, and / or a non-contact sensor module in other modular devices of the modular communication hub 20065 that can be connected to a surgical data network.

[0104] In one aspect, Figure 4The illustrated surgical hub system 20060 may include a combination of a network hub, a network switch, and a network router that connects the device 1a-1n / 2a-2m or the sensing system 20069 to the cloud base system 20064. One or more of the devices 1a-1n / 2a-2m or the sensing system 20069 coupled to the network hub 20061 or the network switch 20062 can collect data in real time and transmit the data to a cloud computer for data processing and operation. It should be understood that cloud computing relies on shared computing resources rather than using local servers or personal devices to process software applications. The term "cloud" can be used as a metaphor for the "Internet", although the term is not so limited. Thus, the term "cloud computing" may be used herein to refer to a "type of Internet-based computing" in which different services (such as servers, storage devices, and applications) are delivered to a modular communication hub 20065 and / or computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space) and devices connected to the modular communication hub 20065 and / or computer system 20063 via the Internet. The cloud infrastructure may be maintained by a cloud service provider. In this case, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services can perform a large number of calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to be connected to a computer that communicates with cloud computing resources and storage devices.

[0105] By applying cloud computer data processing technology to the data collected by the devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical results, reduced costs and improved patient satisfaction. At least some of the devices 1a-1n / 2a-2m can be used to observe tissue status to evaluate the leakage or perfusion of the sealed tissue after tissue sealing and cutting surgery. At least some of the devices 1a-1n / 2a-2m can be used to identify pathology, such as the effects of disease, using cloud-based computing to examine data including images of body tissue samples for diagnostic purposes. This may include positioning and edge confirmation of tissue and phenotype. At least some of the devices 1a-1n / 2a-2m can be used to identify the anatomical structure of the body using various sensors integrated with imaging devices and technologies (such as overlapping images captured by multiple imaging devices). The data (including image data) collected by the devices 1a-1n / 2a-2m can be transmitted to the cloud computing system 20064 or the local computer system 20063 or both for data processing and manipulation, including image processing and manipulation. The data can be analyzed to improve surgical outcomes by determining whether further treatment can be pursued, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions. Such data analysis can be further processed using outcome analysis, and using standardized methods can provide helpful feedback to confirm surgical treatments and surgeon behavior, or suggest modifications to surgical treatments and surgeon behavior.

[0106] Applying cloud computer data processing techniques to the measurement data collected by the sensing systems 20069, the surgical data network can provide improved surgical results, improved recovery results, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 can be used to assess the physiological condition of a surgeon operating on a patient, or a patient preparing for surgery, or a patient recovering after surgery. The cloud-based computing system 20064 can be used to monitor biomarkers associated with the surgeon or patient in real time, and can be used to generate a surgical plan based at least on the measurement data collected prior to surgery, provide control signals to surgical instruments during surgery, and notify patients of complications during the post-operative period.

[0107] The operating room devices 1a-1n can be connected to the modular communication hub 20065 through a wired channel or a wireless channel, depending on the configuration of the devices 1a-1n to the network hub 20061. In one aspect, the network hub 20061 can be implemented as a local network broadcast device that works on the physical layer of the open system interconnection (OSI) model. The network hub can provide connections with devices 1a-1n located in the same operating room network. The network hub 20061 can collect data in the form of packets and send it to the router in half-duplex mode. The network hub 20061 may not store any media access control / Internet protocol (MAC / IP) for transmitting device data. Only one device in the device 1a-1n can send data through the network hub 20061 at a time. The network hub 20061 may not have a routing table or intelligence about where to send information and broadcast all network data on each connection and broadcast all network data to the remote server 20067 of the cloud computing system 20064. Network hub 20061 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose a security risk and cause bottlenecks.

[0108] The operating room devices 2a-2m may be connected to the network switch 20062 via a wired channel or a wireless channel. The network switch 20062 operates in the data link layer of the OSI model. The network switch 20062 may be a multicast device for connecting the devices 2a-2m located in the same operating room to the network. The network switch 20062 may send data to the network router 20066 in the form of a frame and may operate in full-duplex mode. Multiple devices 2a-2m may simultaneously send data via the network switch 20062. The network switch 20062 stores and uses the MAC addresses of the devices 2a-2m to transmit data.

[0109] The network hub 20061 and / or the network switch 20062 may be coupled to a network router 20066 to connect to the cloud computing system 20064. The network router 20066 operates in the network layer of the OSI model. The network router 20066 generates routes for transmitting data packets received from the network hub 20061 and / or the network switch 20062 to cloud-based computer resources to further process and manipulate data collected by any or all of the devices 1a-1n / 2a-2m and the wearable sensing system 20011. The network router 20066 may be employed to connect two or more different networks located in different locations, such as, for example, different operating rooms in the same medical facility or different networks located in different operating rooms in different medical facilities. The network router 20066 may send data to the cloud computing system 20064 in the form of packets and operate in full-duplex mode. Multiple devices may send data simultaneously. The network router 20066 may use IP addresses to transmit data.

[0110] In one example, the network hub 20061 may be implemented as a USB hub that allows multiple USB devices to be connected to a host computer. A USB hub may expand a single USB port to multiple levels so that more ports are available to connect devices to a host system computer. The network hub 20061 may include wired or wireless capabilities for receiving information through a wired channel or a wireless channel. In one aspect, a wireless USB short-range, high-bandwidth wireless radio communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in an operating room.

[0111] In an example, the operating room devices 1a-1n / 2a-2m and / or the sensing system 20069 can communicate with the modular communication hub 20065 via the Bluetooth wireless technology standard for exchanging data from fixed devices and mobile devices over short distances (using short wavelength UHF radio waves from 2.4GHz to 2.485GHz in the ISM band) and building a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or the sensing system 20069 can communicate with the modular communication hub 20065 via a variety of wireless or wired communication standards or protocols, including but not limited to Bluetooth, low-power Bluetooth, near field communication (NFC), Wi-Fi (IEEE802.11 series), WiMAX (IEEE 802.16 series), IEEE 802.20, New Radio (NR), Long Term Evolution (LTE) and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT and its Ethernet derivatives, and any other wireless and wired protocols designated as 3G, 4G, 5G and above. The computing module may include multiple communication modules. For example, the first communication module can be dedicated to shorter-range wireless communications, such as Wi-Fi and Bluetooth Low Energy, Bluetooth Smart, while the second communication module can be dedicated to longer-range wireless communications, such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, etc.

[0112] Modular communication hub 20065 may serve as a central connection for one or more of operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 and may process data types referred to as frames. Frames may carry data generated by devices 1a-1n / 2a-2m and / or sensing systems 20069. When modular communication hub 20065 receives a frame, the frame may be amplified and / or sent to a network router 20066, which may transmit the data to a cloud computing system 20064 or a local computer system 20063 using a plurality of wireless or wired communication standards or protocols, as described herein.

[0113] Modular communication hub 20065 can be used as a standalone device or connected to a compatible network hub 20061 and network switch 20062 to form a larger network. Modular communication hub 20065 is generally easy to install, configure, and maintain, making it a good option for networking operating room devices 1a-1n / 2a-2m.

[0114] Figure 5 A logic diagram of a control system 20220 of a surgical instrument or surgical tool according to one or more aspects of the present disclosure is shown. The surgical instrument or surgical tool may be configurable. The surgical instrument may include a surgical fixation device dedicated to the surgery to be performed, such as an imaging device, a surgical stapler, an energy device, an endoscopic cutter device, etc. For example, the surgical instrument may include any of an electric stapler, a powered stapler generator, an energy device, a pre-energy device, a pre-energy jaw device, an endoscopic cutter clamp, an energy device generator, an operating room imaging system, a smoke evacuator, a suction-irrigation device, an insufflation system, etc. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221, which includes a processor 20222 and a memory 20223. For example, one or more of sensors 20225, 20226, 20227 provide real-time feedback to the processor 20222. The motor 20230 driven by the motor driver 20229 is operably coupled to the longitudinally movable displacement member to drive the I-beam blade element. Tracking system 20228 can be configured to determine the position of the longitudinally movable displacement member. Position information can be provided to processor 20222, which can be programmed or configured to determine the position of the longitudinally movable drive member and the position of the firing member, firing rod and I-beam blade element. Additional motors can be provided at the tool driver interface to control I-beam firing, closure tube travel, shaft rotation and joint movement. Display 20224 can display a variety of operating conditions of the instrument and can include a touch screen function for data input. The information displayed on the display 20224 can be superimposed with an image obtained via an endoscopic imaging module.

[0115] The microcontroller 20221 may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the main microcontroller 20221 may be a LM4F230H5QR ARM Cortex-M4F processor core available from, for example, Texas Instruments, which includes 256KB of single-cycle flash memory or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improving performance above 40MHz, 32KB of single-cycle SRAM, a CPU loaded with Internal ROM for software, 2KB EEPROM, one or more PWM modules, one or more QEI analog and / or one or more 12-bit ADC with 12 analog input channels, details of which can be found in the product datasheet.

[0116] The microcontroller 20221 may include a safety controller including two controller-based families such as TMS570 and RM4x, also known as Hercules ARM CortexR4 produced by Texas Instruments. The safety controller may be configured to be specifically used for IEC 61508 and ISO 26262 safety-critical applications, etc., to provide advanced integrated safety features while delivering scalable execution, connectivity and memory options.

[0117] The microcontroller 20221 can be programmed to perform various functions, such as precise control of the speed and position of the knife and articulation system. In one aspect, the microcontroller 20221 may include a processor 20222 and a memory 20223. The electric motor 20230 may be a brushed DC (DC) motor having a gearbox and a mechanical link to the articulation or knife system. In one aspect, the motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers can be easily replaced for use in a tracking system 20228 including an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, entitled "SYSTEMS AND METHODS FOR CONTROLLING ASURGICAL STAPLING AND CUTTING INSTRUMENT," published on October 19, 2017, which is incorporated herein by reference in its entirety.

[0118] The microcontroller 20221 can be programmed to provide precise control of the speed and position of the displacement member and articulation system. The microcontroller 20221 can be configured to be able to calculate the response in the software of the microcontroller 20221. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used for actual feedback decisions. The observed response can be a favorable tuning value that balances the smooth continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0119] The motor 20230 may be controlled by a motor driver 20229 and may be employed by a firing system of a surgical instrument or tool. In various forms, the motor 20230 may be a brushed DC drive motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver including a field effect transistor (FET). The motor 20230 may be powered by a power assembly that may be releasably mounted to a handle assembly or a tool housing for supplying control power to a surgical instrument or tool. The power assembly may include a battery that may include a plurality of battery cells connected in series that may be used as a power source to provide power to a surgical instrument or tool. In some cases, the battery cells of the power assembly may be replaceable and / or rechargeable. In at least one example, the battery cell may be a lithium-ion battery that may be coupled to the power assembly and detachable from the power assembly.

[0120] The motor driver 20229 may be an A3941 available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal oxide semiconductor field effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may include a unique charge pump regulator that provides full (>10V) gate drive for battery voltages as low as 7V and allows the A3941 to operate at a reduced gate drive as low as 5.5V. A bootstrap capacitor may be used to provide the above-mentioned battery supply voltage required for the N-channel MOSFET. The internal charge pump of the high-side driver may allow DC (100% duty cycle) operation. A diode or synchronous rectification may be used to drive the full bridge in fast decay mode or slow decay mode. In slow decay mode, current recirculation may pass through the high-side FET or the low-side FET. The power FET may be protected from breakdown by a resistor-adjustable dead time. Comprehensive diagnostics provide indications of undervoltage, overtemperature, and power bridge faults, and can be configured to protect the power MOSFETs under most short circuit conditions.Other motor drivers can be easily substituted for use in tracking system 20228 including an absolute positioning system.

[0121] The tracking system 20228 may include a controlled motor drive circuit arrangement including a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system can provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member can represent a longitudinally movable drive member, which includes a rack of drive teeth for meshing engagement with a corresponding drive gear of the gear reducer assembly. In some examples, the displacement member can represent a firing member, which can be adapted and configured to include a rack of drive teeth. In some examples, the displacement member can represent a firing rod or an I-beam, each of which can be adapted and configured to include a rack of drive teeth. Therefore, as used herein, the term displacement member can be generally used to refer to any movable member of a surgical instrument or tool such as a drive member, a firing member, a firing rod, an I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member can be coupled to the firing member, the firing rod, and the I-beam. Therefore, the absolute positioning system can actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member can be coupled to any position sensor 20225 suitable for measuring linear displacement. Therefore, the longitudinally movable drive member, firing member, firing rod or I-beam or a combination thereof can be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor. The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a sliding potentiometer, a magnetic sensing system including a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system including a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system including a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system including a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0122] The electric motor 20230 may include a rotatable shaft operably docked with a gear assembly that is mounted on a displacement member in meshing engagement with a set of drive teeth or a rack of drive teeth. The sensor element may be operably coupled to the gear assembly so that a single rotation of the position sensor 20225 element corresponds to a certain linear longitudinal translation of the displacement member. The arrangement of the transmission and sensor may be connected to a linear actuator via a rack and pinion arrangement, or to a rotary actuator via a spur gear or other connection. The power source may supply power to the absolute positioning system, and the output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member that includes a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable firing member, a firing rod, an I-beam, or a combination thereof.

[0123] A single rotation of the sensor element associated with the position sensor 20225 may be equivalent to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" following a single rotation of the sensor element coupled to the displacement member. The sensor arrangement may be connected via a gear reduction that enables the position sensor 20225 to complete one or more rotations for a full stroke of the displacement member. The position sensor 20225 may complete multiple rotations for a full stroke of the displacement member.

[0124] A series of switches (where n is an integer greater than one) may be employed, alone or in combination with gear reduction, to provide unique position signals for more than one rotation of the position sensor 20225. The state of the switches may be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is provided to the microcontroller 20221. The position sensor 20225 of this sensor arrangement may include an array of magnetic sensors, analog rotation sensors (such as potentiometers), or analog Hall effect elements, which output a unique combination of position signals or values.

[0125] Position sensor 20225 may include any number of magnetic sensing elements, such as, for example, magnetic sensors classified according to whether they measure the total magnetic field or the vector components of the magnetic field. The technology used to produce the above two types of magnetic sensors can cover many aspects of physics and electronics. The technology used for magnetic field sensing may include probe coils, fluxgates, optical pumps, nuclear spins, superconducting quantum interference devices (SQUIDs), Hall effects, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetosensitive diodes, magnetosensitive transistors, optical fibers, magneto-optics, and magnetic sensors based on microelectromechanical systems, etc.

[0126] The position sensor 20225 for the tracking system 20228 including the absolute positioning system may include a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT monolithic magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 20225 interfaces with the microcontroller 20221 to provide an absolute positioning system. The position sensor 20225 may be a low voltage and low power component and may include four Hall effect elements in the area of ​​the position sensor 20225 that may be located above the magnet. A high resolution ADC and an intelligent power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor (also known as the bitwise method and Volder algorithm) may be provided to perform a simple and efficient algorithm to calculate hyperbolic and trigonometric functions, which requires only addition, subtraction, bit shifting, and table lookup operations. The angular position, alarm bit, and magnetic field information may be transmitted to the microcontroller 20221 via a standard serial communication interface, such as a serial peripheral interface (SPI) interface. The position sensor 20225 may provide 12 or 14 bit resolution. The position sensor 20225 may be an AS5055 chip provided in a small QFN 16-pin 4×4×0.85 mm package.

[0127] The tracking system 20228 including an absolute positioning system may include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power source converts the signal from the feedback controller into a physical input to the system: in this case, voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensors may be provided to measure physical parameters of the physical system. In some aspects, one or more other sensors may include sensor arrangements such as those described in U.S. Patent No. 9,345,481, entitled “STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM,” issued on May 24, 2016, which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, entitled “STAPLE CARTRIDGE TISSUETHICKNESS SENSOR SYSTEM,” issued on September 18, 2014, which is incorporated herein by reference in its entirety; and U.S. Patent Application Serial No. 15 / 628,175, entitled “TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT,” filed on June 20, 2017, which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system will have a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuits to combine the calculated response with the measured response using an algorithm (such as weighted average and theoretical control loop) that drives the calculated response toward the measured response. The calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, inductance and resistance to predict the state and output of the physical system by knowing the input.

[0128] Thus, the absolute positioning system can provide the absolute position of the displacement member when the instrument is powered on, and does not retract or advance the displacement member to a reset (zero or home) position as may be required by conventional rotary encoders, which simply count the number of forward or backward steps taken by the motor 20230 to infer the position of the device actuator, drive rod, knife, etc.

[0129] Sensor 20226 (such as, for example, a strain gauge or micro strain gauge) can be configured to measure one or more parameters of the end effector, such as, for example, the amplitude of the strain applied to the anvil during the clamping operation, which can indicate the closing force applied to the anvil. The measured strain can be converted into a digital signal and provided to the processor 20222. Alternatively or in addition to sensor 20226, sensor 20227 (such as, for example, a load sensor) can measure the closing force applied to the anvil by the closed drive system. Sensor 20227 (such as, for example, a load sensor) can measure the firing force applied to the I-beam during the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sliding member, which is configured to enable the nail driver to cam upward to push the nail out to deform and contact the anvil. The I-beam may also include a sharp cutting edge, which can be used to cut tissue when the I-beam is pushed distally by the firing rod. Alternatively, a current sensor 20231 can be employed to measure the current drawn by the motor 20230. For example, the force required to advance the firing member can correspond to the current drawn by the motor 20230. The measured force can be converted into a digital signal and provided to the processor 20222.

[0130] For example, a strain gauge sensor 20226 can be used to measure the force applied to the tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force on the tissue processed by the end effector. A system for measuring the force applied to the tissue grasped by the end effector may include a strain gauge sensor 20226, such as a micro strain gauge, which can be configured to measure, for example, one or more parameters of the end effector. In one aspect, the strain gauge sensor 20226 can measure the amplitude or magnitude of the strain applied to the jaw member of the end effector during the clamping operation, which can indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 20222 of the microcontroller 20221. The load sensor 20227 can measure the force used to operate the knife element, for example, to cut the tissue captured between the anvil and the nail magazine. A magnetic field sensor can be used to measure the thickness of the captured tissue. The measurement results of the magnetic field sensor can also be converted into a digital signal and provided to the processor 20222.

[0131] The microcontroller 20221 can characterize the corresponding values ​​of the selected position of the firing member and / or the velocity of the firing member using the measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue measured by the sensors 20226, 20227, respectively. In one embodiment, the memory 20223 can store techniques, formulas, and / or lookup tables that can be used by the microcontroller 20221 in the evaluation.

[0132] The control system 20220 of the surgical instrument or tool may also include wired or wireless communication circuitry to communicate with the surgical hub 20065, such as Figure 4 shown.

[0133] Figure 6 An exemplary surgical system 20280 according to the present disclosure is shown and may include a surgical instrument 20282 that communicates with a console 20294 or a portable device 20296 via a wired and / or wireless connection through a local area network 20292 and / or a cloud network 20293. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein to measure the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in situ loading or multi-fire loading unit (MFLU) that allows the clinician to fire multiple fasteners multiple times without removing the loading unit 20287 from the surgical site to reload the loading unit 20287.

[0134] The first jaw 20291 and the second jaw 20290 can be configured to clamp tissue therebetween, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 20291 can be configured to fire at least one fastener multiple times, or can be configured to include a replaceable multiple firing fastener bin, which includes multiple fasteners (e.g., nails, clamps, etc.) that can be fired more than once before being replaced. The second jaw 20290 may include an anvil that deforms the fastener or otherwise fixes the fastener when the fastener is ejected from the multiple firing fastener bin.

[0135] The handle 20297 may include a motor coupled to the drive shaft to affect the rotation of the drive shaft. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include a button, a switch, a lever, a slider, a touch screen, and any other suitable input mechanism or user interface that can be engaged by a clinician to activate the motor.

[0136] The control interface of the handle 20297 can communicate with the controller 20298 of the handle 20297 to selectively activate the motor to affect the rotation of the drive shaft. The controller 20298 can be disposed within the handle 20297 and configured to receive input from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 can also include a display that can be viewed by a clinician during use of the handle 20297. The display can be configured to display a portion of the adapter or loading unit data before, during, or after firing the instrument 20282.

[0137] The adapter 20285 may include an adapter identification device 20284 disposed therein, and the loading unit 20287 may include a loading unit identification device 20288 disposed therein. The adapter identification device 20284 may communicate with the controller 20298, and the loading unit identification device 20288 may communicate with the controller 20298. It should be understood that the loading unit identification device 20288 may communicate with the adapter identification device 20284, which relays or transmits communications from the loading unit identification device 20288 to the controller 20298.

[0138] The adapter 20285 may also include a plurality of sensors 20286 (one is shown) disposed about it to detect various conditions of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of firings of the adapter 20285, the peak force of the adapter 20285 during firing, the total amount of force applied to the adapter 20285, the peak retraction force of the adapter 20285, the number of pauses of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to the adapter identification device 20284 in the form of data signals. The data signals of the plurality of sensors 20286 may be stored in the adapter identification device 20284, or may be used to update the adapter data stored in the adapter identification device. The data signals of the plurality of sensors 20286 may be analog or digital. The plurality of sensors 20286 may include a force gauge to measure the force applied to the loading unit 20287 during firing.

[0139] The handle 20297 and the adapter 20285 can be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface can be a direct electrical interface (i.e., comprising electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface can be a contactless electrical interface to wirelessly transmit energy and signals therebetween (e.g., inductive transmission). It is also conceivable that the adapter identification device 20284 and the controller 20298 can communicate wirelessly with each other via a wireless connection separate from the electrical interface.

[0140] The handle 20297 may include a transceiver 20283 that is configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., the LAN 20292, the cloud 20293, the console 20294, or the portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 20280. For example, the controller 20298 may transmit instrument data to the console 20294, the instrument data including the serial number of the attachment adapter (e.g., adapter 20285) attached to the handle 20297, the serial number of the loading unit (e.g., loading unit 20287) attached to the adapter 20285, and the serial number of the multi-fire fastener cartridge loaded into the loading unit. Thereafter, the console 20294 may transmit data associated with the attached cartridge, loading unit, and adapter, respectively (e.g., cartridge data, loading unit data, or adapter data) back to the controller 20298. The controller 20298 may display a message on a local instrument display, or transmit the message to the console 20294 or the portable device 20296 via the transceiver 20283 to display the message on the display 20295 or the portable device screen, respectively.

[0141] Figure 7A diagram of a situational awareness surgical system 5100 is illustrated in accordance with at least one aspect of the present disclosure. Data sources 5126 may include, for example, modular devices 5102 (the modular devices may include sensors configured to detect parameters associated with the patient, HCP, and environment and / or the modular devices themselves), databases 5122 (e.g., an EMR database containing patient records), and patient monitoring devices 5124 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor), HCP monitoring devices 35510, and / or environmental monitoring devices 35512. The surgical hub 5104 may be configured to be able to derive contextual information related to the surgical procedure from the data, for example, based on a particular combination of the received data or a particular order in which the data is received from the data sources 5126. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being the subject of the surgery. This ability of some aspects of the surgical hub 5104 to derive or infer information related to the surgical procedure from the received data may be referred to as "situational awareness." For example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104 that derives contextual information related to the surgical procedure from data received and / or surgical planning information received from the edge computing system 35514 or the enterprise cloud server 35516.

[0142] The situational awareness system of the surgical hub 5104 may be configured to be able to derive contextual information from data received from the data source 5126 in a variety of different ways. For example, the situational awareness system may include a pattern recognition system or a machine learning system (e.g., an artificial neural network) that has been trained on training data to associate various inputs (e.g., data from the database 5122, the patient monitoring device 5124, the modular device 5102, the HCP monitoring device 35510, and / or the environmental monitoring device 35512) with corresponding contextual information about the surgical procedure. The machine learning system may be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In an example, the situational awareness system may include a lookup table that stores pre-characterized environmental information about the surgical procedure associated with one or more inputs (or input ranges) corresponding to environmental information. In response to a query using one or more inputs, the lookup table may return corresponding contextual information used by the situational awareness system to control the modular device 5102. In an example, the contextual information received by the situational awareness system of the surgical hub 5104 may be associated with a specific control adjustment or a set of control adjustments for one or more modular devices 5102. In an example, the situational awareness system may include an additional machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 5102 when contextual information is provided as input.

[0143] The surgical hub 5104 incorporating a situational awareness system can provide numerous benefits to the surgical system 5100. One benefit can include improved interpretation of sensed and collected data, which in turn will improve processing accuracy and / or use of the data during the surgical procedure. Returning to the previous example, the situational awareness surgical hub 5104 can determine the type of tissue being operated on; thus, when an unexpectedly high force is detected for the end effector of a closing surgical instrument, the situational awareness surgical hub 5104 can properly ramp up or ramp down the motor speed of the surgical instrument for the tissue type.

[0144] The type of tissue being operated on can affect the adjustment of the compression rate and load threshold of the surgical stapling and cutting instrument for a specific tissue gap measurement value. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic surgery or an abdominal surgery, thereby allowing the surgical hub 5104 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument for the type of tissue.

[0145] The type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the type of surgery. Since one type of surgery can often be performed in a particular body cavity, the surgical hub 5104 can then control the motor rate of the smoke evacuator appropriately for the body cavity in which the operation is being performed. Thus, the situational awareness surgical hub 5104 can provide consistent smoke evacuation for both thoracic and abdominal surgeries.

[0146] The type of surgery being performed can affect the optimal energy level at which an ultrasonic surgical instrument or radio frequency (RF) electrosurgical instrument operates. For example, arthroscopic surgery may require a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in a fluid. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (e.g., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The situational awareness surgical hub 5104 can determine the type of surgical procedure being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile of the surgical procedure. In addition, the situational awareness surgical hub 5104 can be configured to be able to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the surgical procedure rather than just on a surgery-by-surgery basis. The situational awareness surgical hub 5104 can determine the step of a surgical procedure being performed or to be subsequently performed and then update the control algorithm of the generator and / or ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type based on the surgical step.

[0147] In an example, data may be extracted from additional data sources 5126 to improve conclusions extracted by the surgical hub 5104 from one data source 5126. The situational awareness surgical hub 5104 may augment the data it receives from the modular device 5102 with contextual information about the surgical procedure that has been constructed from other data sources 5126. For example, the situational awareness surgical hub 5104 may be configured to be able to determine whether hemostasis has occurred (e.g., whether bleeding at a surgical site has stopped) based on video or image data received from a medical imaging device. The surgical hub 5104 may be further configured to be able to compare physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device communicatively coupled to the surgical hub 5104) to determine the integrity of the staple line or tissue weld. The situational awareness system of the surgical hub 5104 may take into account the physiological measurement data to provide additional context when analyzing the visual data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0148] For example, if the situational awareness surgical hub 5104 determines that a subsequent step of the surgery requires the use of an RF electrosurgical instrument, it can actively activate the generator connected to the instrument. Actively activating the energy source can allow the instrument to be ready for use as soon as the previous step of the surgery is completed.

[0149] The situational awareness surgical hub 5104 may determine whether the current or subsequent steps of the surgical procedure require a different view or degree of magnification on the display based on the features that the surgeon anticipates needing to view at the surgical site. The surgical hub 5104 may proactively change the displayed view (e.g., provided by a medical imaging device for the visualization system) accordingly, such that the display is automatically adjusted throughout the surgical procedure.

[0150] The situational awareness surgical hub 5104 can determine which step of the surgical procedure is being performed or will be performed next and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the step of the surgical procedure being performed without having to wait for the surgeon to request that specific information.

[0151] Errors may be checked during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 5104 may determine whether the operating room is correctly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 may be configured to be able to determine the type of surgical procedure being performed, retrieve the corresponding list, product location, or setup requirements (e.g., from a memory), and then compare the current operating room layout with the standard layout determined by the surgical hub 5104 for the type of surgical procedure being performed. In some examples, the surgical hub 5104 may compare a list of items for surgery and / or a list of devices paired with the surgical hub 5104 with a suggested or expected list of items and / or devices for a given surgical procedure. If there is any discontinuity between the lists, the surgical hub 5104 may provide an alert indicating that a particular modular device 5102, a patient monitoring device 5124, an HCP monitoring device 35510, an environmental monitoring device 35512, and / or other surgical items are missing. In some examples, the surgical hub 5104 can determine the relative distance or position of the modular device 5102 and the patient monitoring device 5124, for example, via a proximity sensor. The surgical hub 5104 can compare the relative position of the devices to a suggested or expected layout for a particular surgical procedure. If there are any discontinuities between the layouts, the surgical hub 5104 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the suggested layout.

[0152] The situational awareness surgical hub 5104 may determine if a surgeon (or other HCP) is making an error or otherwise deviating from an expected course of action during a surgical procedure. For example, the surgical hub 5104 may be configured to determine the type of surgical procedure being performed, retrieve a corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the surgical procedure to the expected steps or equipment determined by the surgical hub 5104 for the type of surgical procedure being performed. The surgical hub 5104 may provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0153] The surgical instruments (and other modular devices 5102) can be adjusted for the specific context of each surgical procedure (such as adjusting to different tissue types) and verification actions during the surgical procedure. The next steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the surgical room according to the specific context of the surgery.

[0154] Surgical autonomous systems, apparatus, and methods may include aspects of integration with other medical devices, data sources, processes, and mechanisms. For example, surgical autonomous systems, apparatus, and methods may include aspects of integration with a computer-implemented interactive surgical system and / or one or more elements of a computer-implemented interactive surgical system. As described herein, surgical system, surgical autonomous system, and autonomous surgical system are interchangeable.

[0155] refer to Figure 8 , which may provide an overview of the surgical autonomous system 48000. Surgical instrument A 48010 and / or surgical instrument B 48020 may be used in surgical procedures as part of the surgical system 48000. The surgical hub 48035 may be configured to coordinate the flow of information to displays of surgical instruments. For example, the surgical hub may be described in U.S. Patent Application Publication No. US2019-0200844A1 (U.S. Patent Application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. For example, exemplary surgical instruments suitable for use with surgical system 48000 are described under the heading “Surgical Instrument Hardware” in U.S. patent application publication No. US2019-0200844 A1 (U.S. patent application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety.

[0156] Figure 8 An example of a surgical autonomous system 48000 is shown. System 48000 can be used to perform surgery on a patient lying flat on an operating table in a surgical operating room. The robotic system can be used in a surgical procedure as part of a surgical system. For example, the robotic system may be described in U.S. Patent Application Publication No. US2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. The robotic hub can be used to process images of the surgical site for subsequent display to the surgeon via the surgeon's console.

[0157] Other types of robotic systems may be readily adapted for use with the surgical system 48000. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. patent application Ser. No. US2019-0201137A1, filed Dec. 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL” (U.S. Patent Application Ser. No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0158] Various examples of cloud-based analytics performed by the cloud and suitable for use with the present disclosure are described in U.S. patent application publication No. US2019-0206569 A1, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” filed on December 4, 2018 (U.S. patent application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0159] In various aspects, an imaging device can be used in a surgical system and can include at least one image sensor and one or more optical components. Suitable image sensors can include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0160] The optical components of the imaging device may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0161] One or more illumination sources may be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescence spectrum) is that portion of the electromagnetic spectrum that is visible to (e.g., detectable by) the human eye and may be referred to as "visible light" or simply "light." A typical human eye will respond to wavelengths in the range of about 380 nm to about 750 nm in air.

[0162] The invisible spectrum (e.g., non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0163] In various aspects, the imaging device can be configured for use in minimally invasive surgery. Examples of imaging devices suitable for the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophago-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0164] The imaging device can use multispectral monitoring to distinguish the topography and the underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. US2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY" filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. After completing the surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site. It is self-evident that during any surgical operation, the operating room and surgical equipment need to be strictly sterilized. The strict sanitation and sterilization conditions required in the "surgical room" (i.e., operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including imaging devices and their attachments and components. It should be understood that the sterile field can be considered as a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered as the area around the patient prepared for surgery. The sterile field can include properly dressed scrubbed team members, as well as all equipment and fixtures in the area.

[0165] Surgical instrument A 48010 and / or surgical instrument B 48020 may include one or more capabilities. (For example, surgical instrument A includes capabilities B, Z, and D, and surgical instrument includes capabilities C, F, and E). These capabilities may be associated with features that the surgical instrument is able to perform (for example, autonomous function A 48015 associated with surgical instrument A and autonomous function B 48025 associated with surgical instrument B). Examples of these capabilities may be described under the title "Surgical Instrument Hardware" in U.S. patent application publication No. US 2019-0200844A1 (U.S. patent application No. 16 / 209,385), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed on December 4, 2018, the disclosure of which is incorporated herein by reference in its entirety. For example, if the surgical instrument is an endoscopic cutter, one of the capabilities may be the removal of tissue (for example, tissue surrounding the colon when a surgeon is performing a proctocolectomy). These capabilities may include, for example, the removal of tissue relative to a surgical instrument. Fig. 9 The surgical tasks described. For example, the ability to remove tissue may include controlling energy source, cutting, suturing, knob orientation, body orientation, body position, anvil jaw force, reload alignment slot management, etc.

[0166] Surgical instruments A and / or B may be associated with control feedback or fault mitigation feedback, as described herein. For example, the autonomy level with the least manual input may be associated with control feedback. For example, the autonomy level with increased manual input may be associated with fault mitigation feedback.

[0167] The monitoring module 48035 may be generated based on the performance of surgical instruments A and / or B (e.g., generated by a monitoring module located at the surgical hub 48035 or by a monitoring module such as a monitoring module located at the surgical hub 48035). Fig.11 The data may be related to the performance of the current level of autonomy of the surgical instrument. For example, the data may be associated with physical measurements, physiological measurements, etc., such as relative to Fig.10 and Fig.11 These measurements are described in more detail in U.S. Patent Application No. 17 / 156,28, filed on November 10, 2021, under the title “Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements,” the disclosure of which is incorporated herein by reference in its entirety.

[0168] For example, an indication 48045 of the data may be transmitted to a surgical hub 48035 where the indication may be evaluated. In an example, the indication 48045 may be transmitted to a cloud service (e.g., Amazon Web Services) as described herein. The data may be used as an input in an analysis module to, for example, check whether the performance of the surgical instrument falls within a satisfactory range (e.g., exceeds a fault threshold 48040). The indication may be associated with a switch from use control feedback 48005 to fault mitigation feedback.

[0169] Fig. 9 The relationship between control feedback 48060 and fault mitigation feedback 48095 is shown. Fig. 9 As shown, the device may include an expert model 48055. The expert model 48055 may provide parameters so that the surgical instrument can operate autonomously. For example, the surgical instrument can autonomously perform (e.g., be able to perform) one or more surgical tasks. For example, the expert model 48055 may be linked to a machine learning algorithm. As described herein, the machine learning algorithm may include a neural network structure that can generate parameters (e.g., weights in the parameters) to be used on a script, such as located inside the surgical instrument. In an example, the script may be executed remotely, such as at a surgical hub. The parameters output by the expert model 48055 and executed by the script may be dynamic. For example, relative to Figure 6 As described above, the script can be used to execute one or more actuators located on the surgical instrument. For example, the one or more actuators can be linear actuators and / or rotary actuators.

[0170] These actuators can be servo motors located on the surgical instrument. The script can be generated using a machine learning algorithm using adjustable parameters and can be used in conjunction with hardware (such as linear actuators and rotary actuators) to allow the surgical instrument to perform a surgical task. For example, the script can generate control feedback 48060, which can be used as an input to the actuator. The actuator can allow the surgical instrument to be controlled autonomously via the script. For example, the actuator can receive adjustable parameters as input and use these parameters to autonomously perform a surgical task.

[0171] Physical reality measurements 48085 may be obtained. For example, physical reality measurements 48085 may be related to one or more sensors placed on a surgical instrument performing a surgical task. In an example, physical reality measurements 48085 may be obtained from and / or generated from a patient wearing a wearable sensor that measures the patient's biomarkers. In an example, physical reality measurements 48085 may be generated from a surgeon wearing a wearable sensor that measures the surgeon's biomarkers and other health markers.

[0172] The physical reality measurements 48085 may be sent to a surgical hub and may be analyzed at the surgical hub. In an example, the physical reality measurements 48085 may be generated locally on a surgical instrument, for example, by a surgical instrument processor, as described herein. In an example, the surgical instrument and / or the surgical hub may include an analysis module that may analyze the physical reality measurements 48085. If the physical reality measurements 48085 are generated by a remote source such as a surgical hub, the physical reality measurements 48085 may be sent to the surgical instrument via a message. In an example, the message may be delivered via a surgical application program interface (API). The physical reality measurements 48085 (or in other words, data related to these physical reality measurements) may be compared to model measurements (e.g., model data), as described herein.

[0173] Physical reality measurements 48085 may be converted into measured reality data 48075. For example, physical reality measurements 48085 may be delivered by an analysis module located at a surgical hub or locally at a surgical instrument. The output of the analysis module may be data related to the measured reality. For example, physical reality measurements 48085 may be raw data related to the position of a linear actuator and / or a rotary actuator of a surgical instrument. For example, the data may be in the form of a voltage reading. In an example, an analog-to-digital converter (ADC) may be included, and the analog-to-digital converter (ADC) may transform the voltage reading into a bit stream, which may be used as physical reality measurements 48085. The raw data may be sent to an analysis module located at a surgical hub or locally at a surgical instrument, and the analysis module may output measured reality data 48075. When compared to the raw physical reality measurements 48085, the measured reality data 48075 may be in a form more suitable for comparison.

[0174] The measured reality data 48075 may be compared to data associated with the expert model. For example, the expert model may include a data structure that stores model data. The model data may be referred to herein as model reality data. The model reality data may be compared to the measured reality data 48075, and a difference 48070 between the two may be generated. The difference 48070 between the model reality data and the measured reality data 48075 may be used as control feedback 48065 for adjusting parameters that control servo motors, such as the linear actuators and rotary actuators described herein.

[0175] For example, the difference 48070 between the model reality data and the measured reality data may indicate that the speed at which the rotary actuator is rotating is too high. The measured reality data 48075 associated with the rotational speed may be greater than the model reality data associated with the rotational speed of the rotary actuator A positioned at a particular point on the surgical instrument. Based on this indication, a control feedback 48065 may be generated in which a parameter associated with the speed of the rotary actuator A may be reduced so that the speed of the rotary actuator is reduced.

[0176] For example, as long as the difference 48070 between the model reality measurement data and the measured reality data 48075 falls within a threshold, the expert model 48055 can be used to generate control feedback 48065 for controlling the actuator of the autonomously operating surgical instrument. If the difference 48070 between the two does not fall within the threshold 48115 (e.g., exceeds the threshold 48115), the fault mitigation model 48090 can be used (e.g., grounding the control loop to zero 48110 and releasing the fault mitigation loop from grounding to zero 48105).

[0177] The fault mitigation model 48090 can generate parameters to be used by the surgical instrument operating autonomously. These parameters can be set to values ​​that are not associated with risk. In an example, the fault mitigation model 48090 can transition the surgical instrument from an autonomous setting to a manual setting. The surgical instrument using the fault mitigation parameters can control linear actuators and / or rotary actuators and perform other motor controls based on the fault mitigation feedback 48095.

[0178] Physical reality measurements 48085 may be generated to evaluate the performance of fault mitigation control 48100. Similar to expert model 48055, physical reality measurements 48085 may be in the form of raw data and may be passed through an analysis module. The analysis module may generate measured reality data 48075, which may be used and compared to fault model data. Measured reality data 48075 associated with fault mitigation parameters may be compared to fault model reality data, for example, to evaluate the performance of fault mitigation model 48090.

[0179] A notification of switching from expert model 48055 to fault mitigation model 48090 may be sent to a user of the surgical instrument via a message. The message may include parameters set for the surgical instrument. In an example, when the measured reality data 48075 is compared to the model reality data, the patient's heart rate may be compared. This data related to the heart rate may be checked against a threshold 48115. If the heart rate exceeds the threshold, the surgical instrument may switch from using the expert model script to the fault mitigation model script. Fault mitigation model 48090 may generate fault mitigation parameters. The parameters may be known so that the surgical instrument can perform the surgical task without involving risk.

[0180] Fig.10 An example of performing autonomous functions using an ideal model 48120 and a fault model 48145 is shown. Fig.10 As shown, the ideal model 48120 may include one or more surgical tasks. The ideal model 48120 and the expert model may be used interchangeably herein. The ideal model 48120 may be generated by a surgical hub. In an example, the ideal model 48120 may be generated by a surgical hub such as with respect to Fig. 9 The surgical hub described is generated. The surgical tasks of the ideal model (e.g., each surgical task in the surgical tasks) (e.g., surgical task one 48130, surgical task two and surgical task K 48135) may be associated with one or more ideal metrics. The ideal metric may be associated with the ideal physiological and / or physical metric related to the execution of the surgical task. For example, the surgical task may be performed autonomously. The ideal metric may be related to the physiological and / or physical expected conditions that the surgeon may expect to exist based on the surgical task. For example, a surgical task may be to remove surrounding tissue from an organ such as the colon. The ideal physical condition and / or metric of the surgical task may include a patient's heart rate that is higher than a certain value and lower than a certain value (e.g., within an acceptable range). This may be considered as an ideal heart rate range for moving the colon. In an example, the surgical task may be to remove tissue, which may involve the use of an endoscopic cutter. Removing tissue may include executing one or more instrument capabilities, such as positioning the instrument body, controlling the anvil jaw force, and managing the reloading alignment slot. These capabilities may be performed autonomously. For example, a first autonomous function may be associated with an expert model as described herein (e.g., a software module located in the expert model). To perform the capability of a tissue resection surgical task, the expert model may run a first autonomous function (e.g., a script of the autonomous function). Based on having exceeded a fault mitigation threshold, a second autonomous function may be invoked. The second autonomous function may be associated with the fault mitigation model (e.g., may be a software module located in the fault mitigation model). To perform the capability of a tissue resection surgical task, the expert model may switch from running the first autonomous function (e.g., a script of the first autonomous function) to running the second autonomous function (e.g., a script of the second autonomous function).

[0181] The physical and / or physiological ideal conditions may vary based on the surgical task. For example, surgical task two may have different ideal physiological and / or physical conditions when compared to surgical task one. Physical metrics may include data generated from sensors from one or more actuators, such as relative to Fig. 9 and Fig.12In order for the surgical instrument to perform an autonomous function, the surgical instrument may perform surgical task one 48130, surgical task two, and / or surgical task K 48135. The ideal physical and / or physiological conditions may be compared to measured physiological and / or physical conditions, for example, as shown by corresponding measured autonomous functions 48160, which may be generated 48155 from one or more of: sensors, actuators, data generated from robotic functions, data generated from the patient (such as physiological data), data generated from the surgeon (such as physiological surgeon data).

[0182] Data 48155 generated from a patient and / or surgeon may involve the use of wearable devices to measure biomarkers associated with the patient and / or surgeon. These wearable devices are described in more detail in U.S. patent application No. US17 / 156,28 filed on November 10, 2021, under the title "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0183] This data may be generated 48155 and used as input to collect physiological and / or physical metrics. Data associated with ideal physical and / or physiological metrics may be used and compared to data generated from measured physical and / or physiological conditions (e.g., for each of the surgical tasks associated with the measured autonomous functions 48160, such as surgical task 1 48165, surgical task 2, and / or surgical task K 48170), and a difference between the two may be generated and compared to a threshold 48180, which may occur at the surgical hub or locally at the surgical instrument, such as with respect to Fig. 9 and Fig.12 described.

[0184] Determining a threshold 48180 (e.g., a threshold to which the difference between the ideal physical and / or physiological condition and the measured physical and / or physiological condition is compared) can be generated using a machine learning model 48185, which can include a neural network structure. The neural network structure can use one or more of the ideal physical and / or physiological conditions as parameters when determining an appropriate threshold level 48180. If a threshold level 48180 (which can be referred to as a fault mitigation threshold level 48180, for example, relative to the difference between the ideal physical and / or physiological condition and the measured physical and physiological condition) has been exceeded based on the difference between the ideal physical and / or physiological condition and the measured physical and physiological condition, the fault mitigation threshold level 48180 can be generated using a machine learning model 48185, which can include a neural network structure. When determining an appropriate threshold level 48180, the neural network structure can use one or more of the ideal physical and / or physiological conditions as parameters. Fig.12 As described above), the surgical instrument can switch to using fault model 48145.

[0185] Fault model 48145 may relate to surgical instruments associated with fault mitigation feedback. Fault model 48145 may include physiological and / or physical fault metrics, such as fault physical and / or physiological conditions. Similar to the ideal model, these conditions may be compared to the measured physiological and physical conditions described herein. Determining both the ideal physical and physiological conditions and the fault physical and physiological conditions may involve using machine learning model 48175 as described herein. For example, machine learning model 48175 associated with fault mitigation model 47145 may determine a fault mitigation metric that eliminates or minimizes any risk associated with performing one or more surgical tasks (e.g., surgical tasks 1, 2, and / or K).

[0186] The machine learning model 48125 associated with the ideal model can determine one or more physical and / or physiological metrics associated with the optimized execution of the surgical task. The optimized execution can take into account the costs and / or benefits associated with the surgical task. For example, surgical task one may involve a surgeon moving the patient's colon. During the surgical task, the surgeon's heart rate may have an ideal physiological measurement result. The patient's heart rate may have an ideal physiological measurement result. The position of the instrument may have an ideal physical measurement result, which can be determined by the position of one or more actuators. In an example, the data can be analyzed and a value can be generated. In an example, each ideal data collected from the physiological and / or physical conditions can be compared with the corresponding measured physiological and / or physical conditions. For example, the surgeon's ideal heart rate can be compared with the surgeon's measured heart rate as determined during the autonomously functioning surgical task one. During surgical task one 48130, the patient's heart rate can be compared with the measured heart rate of the patient. In an example, if one of the differences between the ideal model physical condition and the measured model physical condition and / or physiological condition exceeds fault mitigation threshold 48180, the surgical instrument may switch from operating based on control feedback 48140 to operating based on fault mitigation feedback 48141. Fault mitigation feedback 48141 may be relative to Fig. 9 In an example, when switching from surgical task one 48130 to surgical task two, a set of ideal physical and / or physiological metrics may be generated by an ideal model, which may be located on the surgical instrument and / or on the surgical hub.

[0187] In an example, the surgical instrument may pull data from a remote source to generate ideal metrics associated with surgical task two, which may be based on historical data associated with surgical task two. The historical data may be based on historical data for the facility or may be based on general historical data associated with surgical task two. The physical and / or physiological conditions and / or metrics associated with these physical and / or physiological conditions may be manually input by the surgeon or another user of the surgical instrument.

[0188] Fig.11 An example of performing an autonomous function using control feedback and fault mitigation feedback is shown. An overview of performing an autonomous function using control feedback and fault mitigation feedback 48195 may be provided. Autonomous function and autonomous surgical task may be used interchangeably herein.

[0189] At 48200, a first autonomous function may be performed. The first autonomous function may be related to Fig. 9 The expert model may be associated with the surgical instrument. For example, the expert model may output parameters to be used when performing the autonomous function. The autonomous function may perform a surgical task associated with the surgical instrument. The control of the surgical instrument may be controlled by control feedback, such as relative to Fig. 9 and Fig.12 The control feedback may update parameters based on the current performance of the autonomous function.

[0190] Control feedback may be generated when model reality data associated with the expert model is compared to measured reality data generated based on physical and / or physiological measurements associated with performing a surgical task. For example, the physical measurements may include the position of an actuator on a surgical instrument performing a surgical task.

[0191] At 48205, a first autonomous function that may be associated with an expert model may be monitored. For example, one or more outputs may be generated based on the performance of the autonomous function.

[0192] The output may be a physical measurement result and may be in the form of raw data. The output may be fed into an analysis module, which may be at a surgical hub or local to a surgical instrument. The output may include data generated based on situational awareness, for example, surgical background data may be sent to a surgical hub. The data generated based on situational awareness may include any data suitable for characterizing the current performance of the patient in view of the ongoing surgical procedure. For example, such data may include the stage of surgery, the current operating room (OR) layout, patient vital signs, instruments, participating personnel, etc. The surgical hub may use the surgical background data when generating measured reality data. The surgical background data may be sent to the analysis module, for example, together with the physical measurement data.

[0193] The expert model can generate parameters by using a machine learning model. These parameters can be used to perform autonomous functions. When generating parameters to be used for autonomous functions, the machine learning model can take into account one or more of the following: historical data; surgical context; the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs, or the extent to which the one or more parameters have exceeded a fault threshold. Threshold and fault threshold can be used interchangeably in this article. When determining whether the difference between the measured real data and the model real data has exceeded the fault threshold, one or more of the following methods described herein can be applied. The first autonomous function can be monitored (for example, as relative to Fig. 9 As described above, the analysis module may output measured real data, which may be compared with the model real data and may generate a difference between the two. The difference between the two may be compared with a fault threshold.

[0194] At 48210, if it is determined that the fault threshold has been exceeded, the surgical instrument may switch to performing a second autonomous function. The second autonomous function may be associated with a fault mitigation model. The fault mitigation model may generate fault mitigation feedback that may cause the parameters to be adjusted to values ​​that enable the surgical instrument to perform the surgical task without involving risk. In an example, a difference between measured real-world data and model real-world data may be generated for each physical measurement obtained from the expert model. A physical measurement related to the speed of the rotary actuator may be generated. The data may be processed in an analysis module. Measured real-world data related to the speed of the rotary actuator may be output from the analysis module.

[0195] Model data related to the speed of the rotary actuator may be included in the expert model as a data structure, such as in a list. The speed of the rotary actuator from the model data structure may be compared with the actual speed of the rotary actuator in the measured reality data, and a corresponding output may be generated based on the difference between the two and the corresponding output may be compared with a corresponding fault threshold. This may be performed for a plurality of physical measurements, thereby generating corresponding measured reality data, which may be compared with corresponding model reality data. The difference may be compared with a corresponding threshold (e.g., a fault threshold).

[0196] In an example, if one of the respective differences between the measured reality data and the model reality data exceeds the respective threshold, the surgical instrument may switch from the first autonomous function to the second autonomous function. The number of differences that must be exceeded before switching to the second autonomous function may be manually input by the user of the surgical instrument. For example, the surgeon may set three as the number of times the respective difference must exceed the fault threshold before switching to the second autonomous function.

[0197] The second autonomous function may be associated with a fault mitigation function as described herein. The difference between the measured reality data and the model reality data may be a single value. For example, the expert model may include an analysis module that evaluates the difference between each measured reality data and each model reality data, and may derive a value based on the difference between the two. In this case, a single difference from all corresponding measured reality data and model reality data may be compared to a single fault threshold. When the single fault threshold is exceeded, the surgical instrument may switch from the first autonomous function to the second autonomous function to perform the surgical task.

[0198] A fault magnitude value may be generated based on a difference. For example, a fault magnitude value may be based on the extent to which a difference between measured reality data and model reality data exceeds a fault threshold. For example, if a difference between measured reality data and model reality data exceeds a fault threshold by a greater amount, the fault magnitude value may be higher than if the difference exceeds the fault threshold by a smaller amount. The fault magnitude value may be used when generating fault mitigation parameters. For example, the fault mitigation parameters may be dynamic and may work within a range. A higher fault magnitude value may cause the surgical instrument to use parameters that create less risk in completing the surgical task. In an example, the fault magnitude value may be used to switch the surgical instrument from autonomous to manual.

[0199] In an example, a fault magnitude may be used to terminate all operations related to the surgical task. The fault magnitude may be generated based on one or more of the following: the surgical context; the type of the first autonomous function; the type of the second autonomous function; historical data associated with the first autonomous function; the number of outputs that exceed the fault threshold; or the degree to which one or more of the outputs exceed the fault threshold. The output may refer to a difference between measured reality data and model reality data. In an example, instead of generating a difference between the measured reality data and the model reality data, the measured reality data may be compared to a threshold. Based on the comparison, it may be determined whether the threshold is exceeded, and in this case, the surgical instrument may be switched from the first autonomous function to the second autonomous function.

[0200] Weights may be assigned to the outputs of the analysis module (e.g., each of the outputs) and / or the measured real data (e.g., each of the measured real data). The speed of the actuator may be assigned a higher weight when compared to the position of the sensor (e.g., a potentiometer). Assigning weights may be performed by an analysis module locally on the surgical instrument or on a surgical hub. Assigning weights may occur on a remote service such as a cloud service. For example, the cloud service may be an Amazon Web Services lambda function. The weights may be considered when comparing whether the difference between the model real data and the measured real data exceeds a fault threshold. For example, if the difference between the measured real data associated with the speed of the actuator exceeds the fault threshold, the surgical instrument may be more likely to switch to the second autonomous function. If the difference between the model real data and the measured real data associated with the position of the actuator exceeds the fault threshold, the surgical instrument may be less likely to switch to the second autonomous function. The assigned weights may be based on one or more of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed a fault threshold (e.g., the number of differences between measured real-world data and modeled real-world data that exceed a threshold); or the extent to which the difference in outputs exceeds the fault threshold.

[0201] As compared to Fig. 9 As described, if the surgical instrument switches from a first autonomous function associated with an expert model to a second autonomous function associated with a fault mitigation model, a message may be sent to a user of the surgical instrument. The message to the user (e.g., a surgeon) may include a set of recommendations. For example, the recommendation may include which measured real-world data has exceeded a threshold, and may suggest how the autonomous function can use fault mitigation feedback to prevent any complications. In an example, the surgical instrument may switch back to the first autonomous function, which is the autonomous function associated with the expert model. These switches may occur multiple times during the performance of a surgical task. These switches may be based on whether the difference between the measured real-world data and the model real-world data falls within or below a threshold. In an example, a risk assessment model may be included and used when determining whether to switch between autonomous functions.

[0202] Generating a recommendation may be based on the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the difference, the number of outputs exceeding a fault threshold, and / or the degree to which the outputs exceed the fault threshold. The expert model may initialize parameters based on training data it receives prior to performing the surgical task. The training data may be based on one or more of the following: the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding a fault threshold, or the degree to which the one or more outputs exceed the threshold.

[0203] Fig.12 An example of an overview of a surgical instrument 48215 and a surgical hub 48280 with control feedback 48220 and fault mitigation feedback 48221 is shown. Fig.12 As shown, surgical instrument 48215 can communicate with surgical hub 48280. In an example, surgical hub 48280 can be a third party service, such as an edge service on a cloud platform.

[0204] The surgical instrument 48215 may include control feedback 48220 and fault mitigation feedback 48221. The control feedback 48220 may be used when performing an autonomous function associated with the surgical task. For example, if the surgical task is to move the colon, the autonomous function may be to release the colon from surrounding tissue, which may involve the use of an endoscopic cutter. The control feedback 48220 may set parameters for the endoscopic cutter to perform the autonomous function (e.g., release the colon from surrounding tissue), for example, via a message to the surgical instrument 48215. The control feedback 48220 may adjust the one or more parameters (e.g., autonomous parameters).

[0205] These parameters can be used as inputs to actuators, which may include Fig. 9 The one or more actuators described herein. These parameters can be adjusted based on feedback from the machine learning model. For example, an error rate can be measured based on the performance of the autonomous function, and if the error rate exceeds a threshold as evaluated by the machine learning model, feedback can be sent to the surgical instrument to adjust the one or more autonomous parameters.

[0206] Control feedback 48220 may include a memory. The memory may store the one or more current parameters. The memory may be accessed via a Fig. 9 and Fig.10 The control feedback 48220 may include a management module. The management module may be responsible for sending an update message to the memory when the one or more autonomous parameters are updated. The control feedback may include a processor that may be responsible for running instructions for performing autonomous functions.

[0207] The processor may communicate with a memory. The memory may include an instruction memory to which the processor may read or write. Control feedback 48220 may be associated with normal operation of an autonomous function for performing an associated surgical task (e.g., autonomously). Control feedback may have one or more autonomous levels. Switching the autonomous level may cause the one or more autonomous parameters to be adjusted. The autonomous level may be stored in a memory together with the one or more autonomous parameters associated with the autonomous level.

[0208] The control feedback 48220 within the surgical instrument may generate an output. The output may be related to the performance of the autonomous function. The output may include the one or more parameters of the autonomous function. The output may be generated and sent to the surgical hub 48280 via a message. The surgical hub 48280 may perform analysis on the output. The surgical hub 48280 may send a response message. In this case, the surgical instrument 48215 may receive input data from the surgical hub 48280 via the response message. The input data may include an indication to update the one or more autonomous parameters associated with the autonomous function.

[0209] As described herein, the machine learning model may be located within the surgical hub 48280. For example, the machine learning model may be a module within the surgical hub 48280. The surgical hub may include a storage device 48305. In an example, the storage device 48305 may be an off-disk storage device. The management module of the surgical hub 48280 may be responsible for sending data associated with the output data of the control feedback 48220 to the storage device 48305. As described herein, the output data may be associated with the performance of the autonomous function and / or the parameters of the autonomous function. Such data may be stored in the storage device 48305 along with a timestamp of the generation of the output data.

[0210] The management module of the surgical hub 48280 may send such data to the storage device 48305. In an example, an analysis module may be included in the surgical hub 48280. The analysis module may compare the performance of the autonomous function generated by the output module with a threshold value (e.g., a fault mitigation threshold). The analysis module may determine whether the output data exceeds the fault mitigation threshold. In this case, if such data exceeds the fault mitigation threshold, the surgical hub 48280 may send a message to the surgical instrument 48215 to switch from using control feedback 48220 to fault mitigation feedback 48221. Fault mitigation feedback 48221 may be such as relative to Fig. 9 , Fig.10 and Fig.11 Fault mitigation feedback as described.

[0211] Fault mitigation feedback 48221 may include one or more actuators. The one or more actuators may be the same actuators as described with respect to control feedback 48220. Parameters for the one or more actuators may be fault mitigation parameters. Fault mitigation parameters may be set to values ​​that are free of risk or expected to have minimal risk based on the performance of the autonomous function. For example, the fault mitigation parameters may include values ​​that make the success rate of the autonomous function 100%. Fault mitigation parameters may be stored in a memory associated with fault mitigation feedback 48221. When performing the autonomous function, the processor of the surgical instrument may request these values. Fault mitigation feedback may be data generated by a fault mitigation model (e.g., using a script associated with a second autonomous function) that is sent to (e.g., signaling) the hardware of the surgical instrument, for example, to autonomously control the surgical instrument when the surgical instrument performs a surgical task. The generated data may be set to a value that ensures that there is no risk or at least a reduced risk involved in the autonomous performance of the surgical task.

[0212] The fault mitigation parameters may be used as input for the actuator to perform the surgical task at hand. The fault mitigation parameters may be sent from the surgical hub 48280. The performance of the surgical instrument 48215 using the fault mitigation parameters may generate output data associated with the performance, which may be sent to the surgical hub 48280 and analyzed. The analysis module may ensure that no risk is involved in the performance of the autonomous function.

[0213] The surgical hub 48280 may detect that a risk still exists based on the performance of the fault mitigation parameters and / or fault mitigation feedback output. The surgical hub 48280 may adjust the fault mitigation parameters, which may be sent via messages and received as input by the surgical instrument 48215. The fault mitigation parameters may be updated in memory. The management module associated with the fault mitigation feedback 48221 may be responsible for updating the memory. In order to perform autonomous functions using the fault mitigation feedback 48221, the processor may write to or read from the memory associated with the fault mitigation feedback 48221.

[0214] Active monitoring and / or reaction may be based on problems encountered (e.g., automated problem resolution). Autonomous function supervisory monitoring and / or fault mitigation may be provided, which may include monitoring autonomous functions, magnitudes, and / or levels to identify faults in autonomous operations (e.g., a portion of an autonomous function). The importance of the identified fault may be determined based on risk, redundancy, timing of occurrence, magnitude of occurrence, or impact of occurrence. The identified fault may be followed by a fault indication to a user and / or to a system external to the hub system. The indication may enable proactive correction of the problem. The indication may be to a user, facility, maintenance personnel, manufacturer, and / or complaint database. The indication may enable fault mitigation, and the result of the mitigation may be part of an indication to one or more systems and / or users. The mitigation may reduce system performance, for example, to enable completion of a procedure or surgery. The mitigation may shut down the affected portion of the system. The mitigation may include alternatives to alternative means of monitoring and / or controlling medical hardware. For example, the mitigation may allow instrument movement to be paused and / or maintained until the system fault is resolved.

[0215] Reports may be provided that may include automatic documentation of complaints and / or issues. Automatic documentation of complaints and / or issues may include one or more of the following features: forms and / or templates that guide the user's recognition and an assistant that can be used when completing; complaint classification and / or forwarding (e.g., issues directed to the manufacturer and / or issues directed to the facility); machine learning aggregation and sorting to find trends; or automated reporting and / or summarization. Automatic documentation of complaints and / or issues may include the surgical hub being fully aware of whether a device in the operating room has experienced a failure or error condition. This information may be autonomously pulled and pushed to a complaint database for review. This information may be classified in a variety of ways. For example, one way may be to have the hub review the device record and determine the severity of the failure (e.g., DFMEA severity) and use it as a category. One way may be to use the hospital's risk system as the basis for classification.

[0216] Automated recording and / or reporting of faults, their potential causes, associated operating environments, solutions or mitigation, and user interactions or experiences may be due to faults. The system may record (e.g., automatically record) log data (e.g., all log data). The data may be stored in a database associated with the system. In an example, the system may record (e.g., automatically record) log data within a given time period. For example, for a rolling log, the system may log (e.g., automatically log) all data within a given time period (e.g., regardless of other severity, criticality, etc.). As the time period passes, the system may automatically start to rewrite the oldest data first to save memory space. For a liability period, the system may automatically log all data within a given time period (e.g., this may be legally required), and may automatically discard the data after the time period has passed. Already identified data may be exempted from the automatic discard feature. For patient outcomes, for example, log data may be retained from the time of surgery until the patient's outcome has been determined. If an adverse event for the patient is identified, the data may be retained without removal. If no adverse events are identified, the log data may be discarded after a certain period of time following the surgical procedure.

[0217] The backup and / or audit of the post-automation operation can be used to review and / or record the automation function and its impact on the user, surgical steps and / or device results. User monitoring of the reaction and response to the autonomous operation can be provided. Override of the main response can be provided. For example, the steps that the surgeon deviates from can be marked for review. For example, the steps that the system considers risky and the surgeon considers acceptable can be annotated by the user as overrides, and the number of these triggers can be used as triggers for audits (e.g., AI reviews, peer reviews, differences with IFU, etc.). The robotic system can perform the expected surgical steps, and device failures may occur. It can be determined that the robotic system has no failures, which may include logging the failure and / or logging the event. The end-of-operation audit of the autonomous operation can be recorded with a log based on the results of the surgical review, user reaction and / or monitoring to record success, failure, user selection and / or the impact on subsequent steps. For example, the recorded images can be automatically annotated with time overlays and device automation tags to provide context and situational awareness of the operation.

[0218] The control system (e.g., local facilities and / or global master systems) may be alerted to problems encountered. The intended use, instructions for use, and contraindications of the device used during the surgical operation may be provided. The medical device may indicate its intended use, instructions for use, and contraindications in the IFU. During the use of the device (e.g., each device), the system may identify whether a violation of the IFU is expected to occur and may warn the user before the event. If the surgeon deems it appropriate, it may be appropriately set to override the system. For example, the control system may warn the device not to be used on ischemic tissue near the colon. The surgeon's determination may be that the area is the best location for the operation. The surgeon may override the system to proceed. The event may be logged to analyze the event after the operation. The number of firings of the device may be exceeded. It may be determined whether the support material is used for any firing that may affect the total number of firings allowed. The operation may have a total surgical time constraint. For example, if the device has been set for too long, the system may warn that the elapsed time has timed out. If it is decided to use, the surgeon may override. The tissue may not be correctly placed and oriented in the jaws. Tissue placement and thickness may be determined. It may be determined that a counterfeit cartridge has been inserted. It may be ensured that the device is cleaned before the cartridge is reloaded. The materials of the device may be monitored to identify if a patient has allergies before using the device, and / or the surgeon may be alerted to the materials of the device before a surgical procedure. Issues may be aggregated into multiple classifications. Classifications may include priority and / or criticality of failures, user confusion, and / or hardware failures (e.g., product queries). There may be escalations between classifications (e.g., frequency and / or severity). Low-risk hardware issues may have redundant systems as backup. If reuse of the redundant system is initiated, this may be used as a trigger, indicating that the primary system is experiencing a problem. If the redundant system is only occasionally utilized, this may indicate that it is necessary to restart the system or module, which may be considered a standard operation.

[0219] A self-service problem solution for the hub and attached system can be provided. The automation of the image search function can be used within the surgical database to provide solution options. As a confirmation before the device energy is activated and / or sutured to minimize leakage and / or complications, the image captured from the endoscope can be compared with the database (bounce off) to identify healthy tissue vs diseased tissue. In an example, the system can be used to identify the best next step based on the problem to alert the surgeon. For example, if the surgeon has completed the staple line suture and perfusion has occurred, the system can alert the surgeon to the best way to solve the leakage based on the historical surgery of the known tissue type. For example, the hub can provide such suggestions (for example, in the form of SNS messages): suturing can be the best technology to solve the leakage, or the energy mode that minimizes time can provide the best technology. The image can be confirmed to the database for navigation confirmation. For example, when the endoscope navigates to the expected site, the system can compare the image from the endoscope with the known database for confirmation / verification check to confirm that the surgeon is moving in the right direction and warn when it deviates from the route. A network connection solution can be provided. The magnitude of the fault can be automatically defined, and problems related to functional operation and demand on the system can be handled with appropriate responses to the problems experienced. For example, this can be used to deal with a fault in an operating room display. If the current operating room display is considered defective during surgery, the system can transfer the content of the display to an alternative display. In an example, the system can segment information. The system can order defective parts. Maintenance personnel can call in facility personnel to repair the faulty equipment. The system can alert the problem of tracking. Low-risk hardware problems may have redundant systems as backup. Major and minor faults and automated responses of the hub can be provided. For example, the hub may experience problems with video output. The power supply of the system may fail, resulting in a complete loss of video feeds. This can be classified as a major fault condition. Overlay system failures may include software errors. To ensure that the correct input is selected, the hub system can accommodate various forms of output from the visualization system. Control measures can ensure that the correct input (e.g., the input that is running the video) is selected for the output. If the host fails (e.g., it cannot be trusted), the display of the overlay can be stopped. In this case, the operation can be performed on a soft-core or hard-core redundant processor. Overlay system failures may include hardware failures. Redundant hardware may be used to maintain full functionality of both systems and may include a voting system (e.g., similar to a fully redundant FCS). Redundant hardware may provide full video functionality without overlay functionality. For a switching hardware solution, the system may have redundant power supplies. Difficulty in implementing a solution may result in it being an active solution (e.g., too many standards for switching).

[0220] Overlay system failures may include power failures. In an example, a visualization system may be provided. The visualization system may fail. For example, the video may be frozen (e.g., but otherwise appears correct), which may be associated with the highest risk. Failures may be no video, blurry or interrupted video, monochrome video, video color changes (e.g., lowest risk), etc. Video failure mitigation may be provided. For example, a control measure may be to provide sufficient visualization to safely remove the instrument from the patient, after which a different visual platform may be found. The control measure may include a device that can process the raw MIPI sensor output from the visualization system and can present a Bayer pattern on a monitor. For example, a white light Bayer pattern sensor may output the brightness value of a pixel (e.g., one pixel) line by line and frame by frame, regardless of the color of the pixel. A series of red, green, and blue filters on each pixel may filter the incoming white light to provide intensity values ​​for these colors. Image signal processing may use the Bayer pattern signal to calculate missing color data for a pixel position (e.g., each pixel position). This process may be referred to as demosaicing. By acquiring the raw sensor signal, placing the signal in memory, reading the signal from memory when required by the HDMI display, and outputting the signal to the display, a preliminary monochrome image can be generated to allow safe removal of the instrument from the patient. A quick start or archive of the operating use conditions immediately before a failure or restart can be used to restore operation to a previous state. System resource management can be provided.

[0221] The following is a list of numbered embodiments that may or may not be claimed:

[0222] 1. An apparatus for controlling a surgical instrument to perform a surgical task, the apparatus comprising a processor configured to:

[0223] monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task; and

[0224] controlling the surgical instrument according to a first model based on the tracked one or more outputs; and

[0225] In the event that at least one of the tracked one or more outputs exceeds a fault threshold, the surgical instrument is switched from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model.

[0226] Advantageously, these features enable the combination of the device and the surgical instrument to act as an autonomous surgical system that is self-contained and fault-resistant. Many instrument controllers that use any kind of output or feedback to control an instrument will provide control instructions that are a direct function of the tracked output (e.g., intended to keep the output constant, set a parameter to a linear function of the output value, etc.). Therefore, if a very abnormal or unexpected output is detected (usually associated with some kind of fault), the control instructions provided to the instrument based on that output may also be abnormal and may be inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds a fault threshold. The device can be external to the surgical instrument, but it can also be a processor or control circuit contained within the surgical instrument.

[0227] 2. The device according to embodiment 1, wherein the first autonomous function and the second autonomous function are different autonomous functions.

[0228] This can be beneficial because it can allow the function to be changed when a fault is detected so that the surgery can be safely paused, stopped, interrupted or completed. For example, the first automatic function may include a clamping operation, and if a threshold force is exceeded, the second autonomous function may be a jaw opening operation.

[0229] 3. An apparatus according to embodiment 1 or embodiment 2, wherein controlling the surgical instrument according to the first model includes providing control feedback generated by the first model to the surgical instrument, wherein the control feedback is associated with setting and / or updating one or more parameters of the surgical instrument, optionally wherein the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument.

[0230] 4. An apparatus according to embodiment 3, wherein the control feedback is configured to be capable of being used as an input to one or more actuators of the surgical instrument, optionally wherein the control feedback is configured to be capable of setting these parameters via a message to the surgical instrument.

[0231] 5. The device of any one of embodiments 1 to 4, wherein the one or more outputs associated with the first autonomous function and / or the surgical task include measured real-world data derived from one or more of:

[0232] measurements from sensors on the surgical instrument;

[0233] measurements, optionally including heart rate, from wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task; and

[0234] Measurements of the position of one or more actuators on the surgical instrument performing the surgical task.

[0235] 6. The device of any one of embodiments 1 to 5, wherein controlling the surgical instrument according to the first model based on the tracked output comprises any of:

[0236] controlling the surgical instrument based on at least one output; or

[0237] comparing at least one output to an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and controlling the surgical instrument based on the difference.

[0238] 7. The apparatus of any one of embodiments 1 to 6, wherein determining whether at least one of the one or more outputs has exceeded a fault threshold comprises any of the following:

[0239] determining that the at least one output includes measured real data that falls outside a given range; or

[0240] comparing at least one output including measured real-world data to an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and determining that the difference falls outside a given range.

[0241] 8. An apparatus according to any one of embodiments 1 to 7, wherein determining whether at least one of the one or more outputs has exceeded a fault threshold includes determining whether each of a predetermined number of the one or more outputs has exceeded the fault threshold, optionally wherein the predetermined number is manually input by a user of the surgical instrument.

[0242] 9. The device of any one of embodiments 1 to 8, wherein controlling the surgical instrument according to the fault mitigation model comprises one or more of the following:

[0243] generating or setting one or more parameters for the surgical instrument that are known or expected to reduce risk associated with completion of the surgical task;

[0244] sending a message to the user of the surgical instrument;

[0245] reducing the level of autonomy of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting;

[0246] Terminate all operations related to the surgical task; or reduce system performance to enable the surgical task to be completed safely.

[0247] Switching the surgical instrument from performing the first autonomous function to performing the second autonomous function associated with the surgical task may correspond to switching to a less autonomous control, switching an operating parameter of the device (e.g., reducing the speed of the cutter, reversing the cutter, etc.), switching to an algorithm to stop operation (e.g., switching from clamping to opening), or changing the power level of an energy device, etc.

[0248] Advantageously, each of these fault mitigation responses can provide a response to a detected fault during the performance of the surgical task, which can reduce the risk of harm to the patient. For example, terminating all operations related to the surgical task can immediately stop the erroneous autonomous function, thereby preventing the erroneous autonomous function from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation measures described above (which do not simply terminate the operation) can provide benefits that are superior to the original fault mitigation control that simply stops the process immediately. This is because, in various surgical contexts, better clinical results (e.g., reduced harm to the patient or reduced complications) can be achieved by continuing to operate the instrument in a safety-oriented manner compared to terminating the operation of the instrument. For example, a surgical instrument having an end effector for clamping tissue (whose clamping arms are biased into a closed position) can be ensured safe by a controller by setting an actuator to release the jaws of the end effector and thereby release any tissue held therein.

[0249] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamping jaws of the end effector. During "normal" operation (i.e., operation according to the first model), the clamping force feedback may be used to control the speed of an actuator that is configured to be able to advance a knife member and / or a slider assembly. This may include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model may indicate the adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a large force is detected (indicating a larger and more robust tissue type), and slightly reducing the firing speed when a small force is detected (indicating a smaller and more fragile tissue type). However, if it is determined that the force on the jaws has dropped below a fault threshold (e.g., has completely dropped to 0 or almost dropped to 0), this may indicate that the tissue has slipped out of the jaws of the end effector. Therefore, the system may switch to a fault mitigation model that may indicate setting the firing speed to a negative value (corresponding to reversing the firing direction and driving the knife back into its recessed housing) because this is a known (or at least reasonably expected) way to ensure the safety of the instrument by allowing the end effector to be repositioned without a serious risk of injury to the patient or operator. The system may also optionally alert the user to the fact that this is occurring.

[0250] As another non-limiting example, when an unexpectedly high force is detected closing the end effector of a surgical instrument, the system may switch to a fault mitigation mode that may gradually increase or decrease the motor speed of the surgical instrument.

[0251] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply radio frequency (RF) and ultrasonic energy to tissue may be controlled according to a first model while monitoring the impedance level at the end effector electrode. The first model may cause RF energy to be applied to the tissue and may adjust energy delivery parameters (e.g., of a generator) based on the monitored impedance to ensure that energy delivery (e.g., power) at the surgical site remains substantially constant. However, if an unexpected peak in impedance (very high or very low) is detected, this may exceed a fault threshold (which is defined as an impedance range within which the detected impedance is expected to fall during normal operation of the device). Therefore, a switch may be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example may include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent undesired tissue transection.

[0252] 10. The device according to any one of embodiments 1 to 9, wherein:

[0253] The one or more outputs tracked are the first one or more outputs tracked;

[0254] The processor is further configured to: monitor the second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task, and

[0255] Controlling the surgical instrument according to the fault mitigation model is based on the tracked second one or more outputs.

[0256] 11. An apparatus according to any one of embodiments 1 to 10, wherein the processor is further configured to be capable of: generating a fault quantity value based on the one or more outputs when the one or more outputs exceed the fault threshold, and wherein controlling the surgical instrument according to the fault mitigation model includes adjusting one or more parameters of the surgical instrument based on the fault quantity value.

[0257] Advantageously, this intelligently adjusts the response of the device and instrumentation to a fault to be proportional to the magnitude of the fault.

[0258] 12. An apparatus according to embodiment 11, wherein the fault value is generated based on at least one of: the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0259] Advantageously, this enables detection of, for example, faults that may be critical but not numerous (or vice versa), thereby allowing the device to strike a balance between importance and quantity when mitigating faults. Additionally or alternatively, this advantageously allows more important surgical functions (e.g., life-threatening) to receive fault mitigation priority over less important surgical functions (e.g., practicality / surgeon ease of use).

[0260] 13. The apparatus of any one of embodiments 1 to 12, wherein the processor is further configured to:

[0261] performing analysis on the tracked one or more outputs; and

[0262] A comparison output is generated based on the analysis; and wherein the processor is configured to: if the comparison output exceeds the fault threshold, perform the switching from performing the first autonomous function to performing the second autonomous function.

[0263] Advantageously, in this manner, switching of autonomous functionality may be performed in response to relative output issues rather than absolute output issues (e.g., comparable to an ideal, desired, and / or conceptual level of output). Additionally or alternatively, it may be beneficial to perform such functionality switching in response to a relationship between one or more outputs, thereby allowing for greater nuance.

[0264] 14. An apparatus according to embodiment 13, wherein the processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of the following: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0265] 15. The apparatus of any one of embodiments 1 to 14, wherein the processor is further configured to:

[0266] In the event that the one or more outputs exceed the fault threshold, an indication of a fault message is sent to a user of a surgical instrument associated with the surgical task, wherein the fault message includes a fault type and a fault magnitude.

[0267] 16. An apparatus according to embodiment 15, wherein the fault message includes a set of recommendations based on: the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0268] 17. An apparatus according to any one of embodiments 1 to 16, wherein the processor is further configured to be capable of determining the fault threshold and fault mitigation feedback based on training data.

[0269] 18. An apparatus according to embodiment 17, wherein the training data is based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0270] 19. An apparatus according to any one of embodiments 1 to 18, wherein the fault threshold is adjusted based on situational awareness.

[0271] 20. A computer-implemented method for controlling a surgical instrument to perform a surgical task, the method comprising:

[0272] monitoring a first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task;

[0273] providing digital commands or user recommendations for controlling the surgical instrument according to the first model based on the tracked one or more outputs; and

[0274] Based on determining that at least one of the one or more outputs has exceeded a fault threshold, providing a digital command or user recommendation for switching the surgical instrument from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model.

[0275] Advantageously, these features enable the combination of a computer implementing the method and the surgical instrument to act as a self-contained and fault-resistant autonomous surgical system. Many instrument control methods that use any kind of output or feedback to control an instrument involve providing control instructions that are a direct function of the tracked output (e.g., intended to keep the output constant, set a parameter to a linear function of the output value, etc.). Therefore, if a very abnormal or unexpected output is detected (usually associated with some kind of fault), the control instructions provided to the instrument based on that output may also be abnormal and may be inappropriate (or even unsafe). In the present invention, this is avoided because the control model is changed when the tracked output exceeds a fault threshold. The computer implementing the method may be external to the surgical instrument, but it may also be a processor or control circuit contained within the surgical instrument itself.

[0276] 21. A method according to embodiment 20, wherein the first autonomous function and the second autonomous function are different autonomous functions.

[0277] This can be beneficial because it can allow the function to be changed when a fault is detected so that the surgery can be safely paused, stopped, interrupted or completed. For example, the first automatic function may include a clamping operation, and if a threshold force is exceeded, the second autonomous function may be a jaw opening operation.

[0278] 22. A method according to embodiment 20 or embodiment 21, wherein the digital command or the user recommendation for controlling the surgical instrument according to the first model includes one or more values ​​for setting or updating parameters of the surgical instrument, optionally wherein the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument.

[0279] 23. A method according to embodiment 22, wherein the values ​​are suitable for use as inputs to one or more actuators of the surgical instrument, optionally wherein the digital command is a message to the surgical instrument configured to be capable of setting the parameters of the surgical instrument.

[0280] 24. The method of any one of embodiments 20 to 23, wherein the one or more outputs associated with the first autonomous function and / or the surgical task include measured real-world data derived from one or more of:

[0281] measurements from sensors on the surgical instrument;

[0282] measurements, optionally including heart rate, from wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task; and

[0283] Measurements of the position of one or more actuators on the surgical instrument performing the surgical task.

[0284] 25. The method of any one of embodiments 20 to 24, wherein the digital command or the user recommendation for controlling the surgical instrument according to the first model based on the tracked output is any of:

[0285] A digital command or user recommendation for controlling the surgical instrument based on at least one output; or

[0286] Digital commands or user recommendations for controlling the surgical instrument based on a difference determined by comparing at least one output to an ideal output generated by the first model.

[0287] 26. The method of any one of embodiments 20 to 25, wherein determining whether at least one of the one or more outputs has exceeded a fault threshold comprises any of the following:

[0288] determining that the at least one output includes measured real data that falls outside a given range; or

[0289] comparing at least one output including measured real-world data to an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and determining that the difference falls outside a given range.

[0290] 27. A method according to any one of embodiments 20 to 26, wherein determining whether at least one of the one or more outputs has exceeded a fault threshold includes determining whether each of a predetermined number of the one or more outputs has exceeded the fault threshold, optionally wherein the predetermined number is manually entered by a user of the surgical instrument.

[0291] 28. The method of any one of embodiments 20 to 27, wherein the digital command or the user recommendation for switching the surgical instrument from performing the first autonomous function to performing the second autonomous function comprises a digital command or user recommendation for:

[0292] generating or setting one or more parameters for the surgical instrument that are known or expected to reduce risk associated with completion of the surgical task;

[0293] sending a message to the user of the surgical instrument;

[0294] reducing the level of autonomy of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting;

[0295] terminate all operations related to the surgical task; or

[0296] System performance is reduced to enable the surgical task to be completed safely.

[0297] Switching the surgical instrument from performing the first autonomous function to performing the second autonomous function associated with the surgical task may correspond to switching to a less autonomous control, switching an operating parameter of the device (e.g., reducing the speed of the cutter, reversing the cutter, etc.), switching to an algorithm to stop operation (e.g., switching from clamping to opening), or changing the power level of an energy device, etc.

[0298] Advantageously, each of these fault mitigation responses can provide a response to a detected fault during the performance of the surgical task, which can reduce the risk of harm to the patient. For example, terminating all operations related to the surgical task can immediately stop the erroneous autonomous function, thereby preventing the erroneous autonomous function from inadvertently causing any harm (or at least any further harm) to the patient. Alternatively, any of the other fault mitigation measures described above (which do not simply terminate the operation) can provide benefits that are superior to the original fault mitigation control that simply stops the process immediately. This is because, in various surgical contexts, better clinical results (e.g., reduced harm to the patient or reduced complications) can be achieved by continuing to operate the instrument in a safety-oriented manner compared to terminating the operation of the instrument. For example, a surgical instrument having an end effector for clamping tissue (whose clamping arms are biased into a closed position) can be ensured safe by a controller by setting an actuator to release the jaws of the end effector and thereby release any tissue held therein.

[0299] As another non-limiting example, performing the first autonomous function may include controlling the firing of a surgical cutting / stapling instrument while monitoring the force detected between the clamping jaws of the end effector. During "normal" operation (i.e., operation according to the first model), the clamping force feedback may be used to control the speed of an actuator that is configured to be able to advance a knife member and / or a slider assembly. This may include controlling the linear speed of a linear actuator or the rotational speed of a rotary actuator. The first model may indicate the adjustment of the actuator according to the force on the jaws, for example, by slightly increasing the firing speed when a large force is detected (indicating a larger and more robust tissue type), and slightly reducing the firing speed when a small force is detected (indicating a smaller and more fragile tissue type). However, if it is determined that the force on the jaws has dropped below a fault threshold (e.g., has completely dropped to 0 or almost dropped to 0), this may indicate that the tissue has slipped out of the jaws of the end effector. Therefore, the system may switch to a fault mitigation model that may indicate setting the firing speed to a negative value (corresponding to reversing the firing direction and driving the knife back into its recessed housing) because this is a known (or at least reasonably expected) way to ensure the safety of the instrument by allowing the end effector to be repositioned without a serious risk of injury to the patient or operator. The system may also optionally alert the user to the fact that this is occurring.

[0300] As another non-limiting example, when an unexpectedly high force is detected closing the end effector of a surgical instrument, the system may switch to a fault mitigation mode that may gradually increase or decrease the motor speed of the surgical instrument.

[0301] As another non-limiting example, an electrosurgical cutting / sealing instrument configured to supply radio frequency (RF) and ultrasonic energy to tissue may be controlled according to a first model while monitoring the impedance level at the end effector electrode. The first model may cause RF energy to be applied to the tissue and may adjust energy delivery parameters (e.g., of a generator) based on the monitored impedance to ensure that energy delivery (e.g., power) at the surgical site remains substantially constant. However, if an unexpected peak in impedance (very high or very low) is detected, this may exceed a fault threshold (which is defined as an impedance range within which the detected impedance is expected to fall during normal operation of the device). Therefore, a switch may be made from performing a first autonomous function according to the first model to performing a second autonomous function according to a fault mitigation model. Performing the second autonomous function in this example may include switching from providing RF energy at the end effector to providing ultrasonic energy to prevent undesired tissue transection.

[0302] 29. The method according to any one of embodiments 20 to 28, wherein:

[0303] The one or more outputs tracked are the first one or more outputs tracked;

[0304] The method further includes monitoring the second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task, and

[0305] The digital command or the user recommendation for controlling the surgical instrument according to the fault mitigation model is based on the tracked second one or more outputs.

[0306] 30. The method according to any one of embodiments 20 to 29, further comprising:

[0307] In the event that the one or more outputs exceed the fault threshold, a fault magnitude is generated based on the one or more outputs, and wherein the digital command or the user recommendation is to adjust one or more parameters of the surgical instrument based on the fault magnitude.

[0308] Advantageously, this intelligently adjusts the computer and instrument response to a fault to be proportional to the magnitude of the fault.

[0309] 31. A method according to embodiment 30, wherein the fault value is generated based on at least one of: the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0310] Advantageously, this enables detection of, for example, failures that may be critical but not numerous (or vice versa), thereby allowing the computer to strike a balance between importance and quantity when mitigating failures. Additionally or alternatively, this advantageously allows more important surgical functions (e.g., life-threatening) to receive priority mitigation over less important surgical functions (e.g., practicality / surgeon ease of use).

[0311] 32. The method according to any one of embodiments 20 to 31, further comprising:

[0312] performing analysis on the tracked one or more outputs; and

[0313] A comparison output is generated based on the analysis; and wherein, based on determining that the comparison output has exceeded the fault threshold, the digital command or the user recommendation is provided for switching the surgical instrument from performing the first autonomous function to performing the second autonomous function.

[0314] Advantageously, in this manner, switching of autonomous functionality may be performed in response to relative output issues rather than absolute output issues (e.g., comparable to an ideal, desired, and / or conceptual level of output). Additionally or alternatively, it may be beneficial to perform such functionality switching in response to a relationship between one or more outputs, thereby allowing for greater nuance.

[0315] 33. The method according to embodiment 32 further comprises:

[0316] A weight is assigned to each of the one or more tracked outputs based on at least one of: historical data, surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

[0317] 34. The method according to any one of embodiments 20 to 33, further comprising:

[0318] In the event that the one or more outputs exceed the fault threshold, an indication of a fault message is sent to a user of a surgical instrument associated with the surgical task, wherein the fault message includes a fault type and a fault magnitude.

[0319] 35. A method according to embodiment 34, wherein the fault message includes a set of recommendations based on: the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0320] 36. The method according to any one of embodiments 20 to 35, further comprising:

[0321] The fault threshold and fault mitigation feedback are determined based on training data.

[0322] 37. A method according to embodiment 36, wherein the training data is based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs that exceed the fault threshold, or the extent to which the one or more outputs exceed the fault threshold.

[0323] 38. A method according to any one of embodiments 20 to 37, wherein the fault threshold is adjusted based on situational awareness.

[0324] 39. A computer program product for causing a computer to execute the method according to any one of embodiments 20 to 38.

[0325] 40. A non-transitory computer-readable storage medium comprising computer-readable instructions that, when executed by a computer, cause the computer to perform a method according to any one of embodiments 20 to 38.

[0326] Following are numbered aspects of the disclosure that may or may not be claimed.

[0327] 1. A device for monitoring a first autonomous function associated with a surgical task, the device comprising:

[0328] A processor configured to:

[0329] performing the first autonomous function associated with the surgical task, wherein performing the first autonomous function is associated with control feedback;

[0330] monitoring the first autonomous function by tracking one or more outputs associated with the control feedback; and

[0331] In the event that the one or more outputs exceed a fault threshold, switching from executing the first autonomous function to executing a second autonomous function associated with the surgical task, wherein the second autonomous function is associated with fault mitigation feedback.

[0332] 2. The apparatus according to aspect 1, wherein the processor is further configured to:

[0333] generating a fault magnitude based on the one or more outputs if the one or more outputs exceed the fault threshold; and

[0334] One or more parameters associated with the fault mitigation feedback are adjusted based on the fault magnitude.

[0335] 3. An apparatus according to aspect 2, wherein the fault value is generated based on at least one of: the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0336] 4. The apparatus according to aspect 1, wherein the processor is further configured to:

[0337] performing analysis on the tracked one or more outputs associated with the control feedback;

[0338] generating a comparison output based on the analysis; and

[0339] In the event that the comparison output exceeds the fault threshold, switching from executing the first autonomous function to executing the second autonomous function associated with the surgical task, wherein the second autonomous function is associated with the fault mitigation feedback.

[0340] 5. An apparatus according to aspect 4, wherein the processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0341] 6. The apparatus according to aspect 1, wherein the processor is further configured to:

[0342] In the event that the one or more outputs exceed the fault threshold, an indication of a fault message is sent to a user of a surgical instrument associated with the surgical task, wherein the fault message includes a fault type and a fault magnitude.

[0343] 7. An apparatus according to aspect 6, wherein the fault message includes a set of recommendations based on: the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0344] 8. The apparatus of clause 1, wherein the processor is further configured to determine the fault threshold and the fault mitigation feedback based on training data.

[0345] 9. An apparatus according to aspect 8, wherein the training data is based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0346] 10. The apparatus of aspect 1, wherein the fault threshold is adjusted based on situational awareness.

[0347] 11. A method for monitoring a first autonomous function associated with a surgical task, the method comprising:

[0348] performing the first autonomous function associated with the surgical task, wherein performing the first autonomous function is associated with control feedback;

[0349] monitoring the first autonomous function by tracking one or more outputs associated with the control feedback; and

[0350] In the event that the one or more outputs exceed a fault threshold, switching from executing the first autonomous function to executing a second autonomous function associated with the surgical task, wherein the second autonomous function is associated with fault mitigation feedback.

[0351] 12. The method according to aspect 11, further comprising:

[0352] generating a fault magnitude based on the one or more outputs if the one or more outputs exceed the fault threshold; and

[0353] One or more parameters associated with the fault mitigation feedback are adjusted based on the fault magnitude.

[0354] 13. A method according to aspect 12, wherein the fault value is generated based on at least one of: the surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0355] 14. The method according to aspect 11, further comprising:

[0356] performing analysis on the tracked one or more outputs associated with the control feedback;

[0357] generating a comparison output based on the analysis; and

[0358] In the event that the comparison output exceeds the fault threshold, switching from executing the first autonomous function to executing the second autonomous function associated with the surgical task, wherein the second autonomous function is associated with the fault mitigation feedback.

[0359] 15. The method according to aspect 14, further comprising:

[0360] A weight is assigned to each of the one or more tracked outputs based on at least one of: historical data, surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

[0361] 16. The method according to aspect 11, further comprising:

[0362] In the event that the one or more outputs exceed the fault threshold, an indication of a fault message is sent to a user of a surgical instrument associated with the surgical task, wherein the fault message includes a fault type and a fault magnitude.

[0363] 17. A method according to aspect 16, wherein the fault message includes a set of recommendations based on: the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0364] 18. The method according to aspect 11, further comprising:

[0365] The fault threshold and fault mitigation feedback are determined based on training data.

[0366] 19. A method according to aspect 18, wherein the training data is based on at least one of: historical data, surgical context, the type of the first autonomous function, the type of the second autonomous function, historical data associated with the first autonomous function, the number of outputs exceeding the fault threshold, or the degree to which the one or more outputs exceed the fault threshold.

[0367] 20. The method of clause 11, wherein the fault threshold is adjusted based on situational awareness.

Claims

1. An apparatus for controlling a surgical instrument to perform a surgical task, the apparatus comprising a processor configured to: monitoring the first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task; as well as controlling the surgical instrument according to a first model based on the tracked one or more outputs; as well as In the event that at least one of the tracked one or more outputs exceeds a fault threshold, the surgical instrument is switched from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model.

2. The device according to claim 1, wherein: The first autonomous function and the second autonomous function are different autonomous functions.

3. The device according to claim 1 or claim 2, wherein: Controlling the surgical instrument according to the first model includes providing control feedback generated by the first model to the surgical instrument, wherein the control feedback is associated with setting and / or updating one or more parameters of the surgical instrument, optionally wherein the one or more parameters of the surgical instrument include parameters of a script for controlling the surgical instrument.

4. The device according to claim 3, wherein: The control feedback is configured to be usable as an input to one or more actuators of the surgical instrument, optionally wherein the control feedback is configured to be capable of setting the parameter via a message to the surgical instrument.

5. The device according to any one of claims 1 to 4, wherein: The one or more outputs associated with the first autonomous function and / or the surgical task include measured reality data derived from one or more of: measurements from sensors on the surgical instrument; measurements, optionally including heart rate, from wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task; and Measurements of the position of one or more actuators on the surgical instrument performing the surgical task.

6. The device according to any one of claims 1 to 5, wherein: Controlling the surgical instrument according to the first model based on the tracked output includes any of: controlling the surgical instrument based on at least one output; or comparing at least one output to an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; and controlling the surgical instrument based on the difference.

7. The device according to any one of claims 1 to 6, wherein: Determining whether at least one of the one or more outputs has exceeded a fault threshold comprises any of the following: determining that the at least one output includes measured real data that falls outside a given range; or comparing at least one output including measured real-world data to an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; And determining that the difference falls outside a given range.

8. The device according to any one of claims 1 to 7, wherein: Determining whether at least one of the one or more outputs has exceeded a fault threshold comprises determining whether each of a predetermined number of the one or more outputs has exceeded the fault threshold, optionally wherein the predetermined number is manually input by a user of the surgical instrument.

9. The device according to any one of claims 1 to 8, wherein: Controlling the surgical instrument according to the fault mitigation model includes one or more of the following: generating or setting one or more parameters for the surgical instrument that are known or expected to reduce risk associated with performance of the surgical task; sending a message to the user of the surgical instrument; reducing the level of autonomy of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting; Terminate all operations related to the surgical task; or reduce system performance to enable the surgical task to be completed safely.

10. The device according to any one of claims 1 to 9, wherein: The one or more outputs tracked are the first one or more outputs tracked; The processor is further configured to: monitor the second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task, and Controlling the surgical instrument according to the fault mitigation model is based on the tracked second one or more outputs.

11. The device according to any one of claims 1 to 10, wherein: The processor is further configured to be capable of: generating a fault magnitude based on the one or more outputs when the one or more outputs exceed the fault threshold, and wherein controlling the surgical instrument according to the fault mitigation model includes adjusting one or more parameters of the surgical instrument based on the fault magnitude.

12. The device according to claim 11, wherein The fault magnitude is generated based on at least one of: surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs exceeding the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

13. The device according to any one of claims 1 to 12, wherein: The processor is further configured to: performing analysis on the tracked one or more outputs; and A comparison output is generated based on the analysis; and wherein the processor is configured to: perform the switching from performing the first autonomous function to performing the second autonomous function if the comparison output exceeds the fault threshold.

14. The device according to claim 13, wherein: The processor is further configured to assign a weight to each of the one or more tracked outputs based on at least one of: historical data, surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs exceeding the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

15. The device according to any one of claims 1 to 14, wherein: The processor is further configured to: In the event that the one or more outputs exceed the fault threshold, an indication of a fault message is sent to a user of a surgical instrument associated with the surgical task, wherein the fault message includes a fault type and a fault magnitude.

16. The device according to claim 15, wherein: The fault message includes a set of recommendations based on: a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

17. The device according to any one of claims 1 to 16, wherein: The processor is further configured to determine the fault threshold and fault mitigation feedback based on training data.

18. The device according to claim 17, wherein: The training data is based on at least one of: historical data, surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

19. The device according to any one of claims 1 to 18, wherein: The fault threshold is adjusted based on situational awareness.

20. A computer-implemented method for controlling a surgical instrument to perform a surgical task, the method comprising: monitoring the first autonomous function associated with the surgical task by tracking one or more outputs associated with the first autonomous function and / or the surgical task; providing digital commands or user recommendations for controlling the surgical instrument according to a first model based on the tracked one or more outputs; as well as Based on determining that at least one of the one or more outputs has exceeded a fault threshold, providing a digital command or user recommendation for switching the surgical instrument from performing the first autonomous function to performing a second autonomous function associated with the surgical task by controlling the surgical instrument according to a fault mitigation model.

21. The method according to claim 20, wherein: The first autonomous function and the second autonomous function are different autonomous functions.

22. The method of claim 20 or claim 21, wherein: The digital command or the user recommendation for controlling the surgical instrument according to the first model comprises one or more values ​​for setting or updating parameters of the surgical instrument, optionally wherein the one or more parameters of the surgical instrument comprise parameters of a script for controlling the surgical instrument.

23. The method according to claim 22, wherein: The value is suitable for use as an input to one or more actuators of the surgical instrument, optionally wherein the digital command is a message to the surgical instrument, the message being configured to enable setting of the parameter of the surgical instrument.

24. The method according to any one of claims 20 to 23, wherein: The one or more outputs associated with the first autonomous function and / or the surgical task include measured reality data derived from one or more of: measurements from sensors on the surgical instrument; measurements, optionally including heart rate, from wearable sensors that measure biomarkers or other health markers of the patient or surgeon associated with the surgical task; and Measurements of the position of one or more actuators on the surgical instrument performing the surgical task.

25. The method according to any one of claims 20 to 24, wherein: The digital command or the user recommendation for controlling the surgical instrument according to the first model based on the tracked output is any of: digital commands or user recommendations for controlling the surgical instrument based on at least one output; or A digital command or user recommendation for controlling the surgical instrument based on a difference determined by comparing at least one output to an ideal output generated by the first model.

26. The method according to any one of claims 20 to 25, wherein: Determining whether at least one of the one or more outputs has exceeded a fault threshold comprises any of the following: determining that the at least one output includes measured real data that falls outside a given range; or comparing at least one output including measured real-world data to an ideal output generated by the first model to determine a difference between the at least one output and the ideal output; And determining that the difference falls outside a given range.

27. The method according to any one of claims 20 to 26, wherein: Determining whether at least one of the one or more outputs has exceeded a fault threshold comprises determining whether each of a predetermined number of the one or more outputs has exceeded the fault threshold, optionally wherein the predetermined number is manually input by a user of the surgical instrument.

28. The method according to any one of claims 20 to 27, wherein: The digital command or the user recommendation for switching the surgical instrument from performing the first autonomous function to performing the second autonomous function comprises a digital command or user recommendation for: generating or setting one or more parameters for the surgical instrument that are known or expected to reduce risk associated with performance of the surgical task; sending a message to the user of the surgical instrument; reducing the level of autonomy of the surgical instrument and / or switching the surgical instrument from an autonomous setting to a manual setting; terminate all operations related to the surgical task; or System performance is reduced to enable the surgical task to be completed safely.

29. A method according to any one of claims 20 to 28, wherein: The one or more outputs tracked are the first one or more outputs tracked; The method further includes monitoring the second autonomous function associated with the surgical task by tracking a second one or more outputs associated with the second autonomous function and / or the surgical task, and The digital commands or the user recommendations for controlling the surgical instrument according to the fault mitigation model are based on the tracked second one or more outputs.

30. The method according to any one of claims 20 to 29, further comprising: In the event that the one or more outputs exceed the fault threshold, a fault magnitude is generated based on the one or more outputs, and wherein the digital command or the user recommendation is to adjust one or more parameters of the surgical instrument based on the fault magnitude.

31. The method according to claim 30, wherein: The fault magnitude is generated based on at least one of: surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs exceeding the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

32. The method according to any one of claims 20 to 31, further comprising: performing analysis on the tracked one or more outputs; as well as generating a comparison output based on the analyzing; And wherein, based on determining that the comparison output has exceeded the fault threshold, providing the digital command or the user recommendation for switching the surgical instrument from performing the first autonomous function to performing the second autonomous function.

33. The method of claim 32, further comprising: Assigning a weight to each of the one or more tracked outputs based on at least one of: historical data, surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

34. The method according to any one of claims 20 to 33, further comprising: In the event that the one or more outputs exceed the fault threshold, an indication of a fault message is sent to a user of a surgical instrument associated with the surgical task, wherein the fault message includes a fault type and a fault magnitude.

35. The method of claim 34, wherein: The fault message includes a set of recommendations based on: a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

36. The method according to any one of claims 20 to 35, further comprising: The fault threshold and fault mitigation feedback are determined based on training data.

37. The method of claim 36, wherein: The training data is based on at least one of: historical data, surgical context, a type of the first autonomous function, a type of the second autonomous function, historical data associated with the first autonomous function, a number of outputs that exceed the fault threshold, or an extent to which the one or more outputs exceed the fault threshold.

38. The method according to any one of claims 20 to 37, wherein: The fault threshold is adjusted based on situational awareness.

39. A computer program product for causing a computer to execute the method according to any one of claims 20 to 38.

40. A non-transitory computer-readable storage medium comprising computer-readable instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 20 to 38.

Images