Peer-to-peer surgical instrument monitoring

Abstract

Systems, methods, and means may be provided for a smart surgical instrument or device to monitor other surgical instruments or devices in a peer-to-peer interconnected surgical ecosystem. The monitoring and / or recording may be performed using a peer-to-peer surgical ecosystem without involving a central computing device. A surgical instrument based on its respective capabilities may configure itself as a monitoring surgical instrument and other surgical instruments that are part of the surgical ecosystem as peer surgical instruments. A monitoring surgical instrument may establish a peer-to-peer connection with a peer surgical instrument to monitor and record surgical tasks on the second surgical instrument. The monitoring surgical instrument may use the established peer-to-peer connection to begin monitoring and / or recording surgical data associated with surgical tasks being performed on the peer surgical instrument.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is related to the following concurrently filed applications, the contents of each of which are incorporated herein by reference: Attorney Docket No. END9438USNP1, concurrently filed, titled "A METHOD FOR ADVANCED ALGORITHM SUPPORT." [Background technology]

[0002] Patient care, in general, improves when it is tailored to the individual. Because every person has different needs, surgical and interventional solutions that center every patient's unique journey can represent an efficient and innovative path to healing. At the same time, the high stakes of patient care, particularly the surgical process, often focus on conservative, repeatable activities.

[0003] For example, innovative medical technologies such as advanced surgical assistance computing systems and intelligent surgical instruments may improve approaches to patient care and address the specific needs of healthcare providers.

[0004] The ever-increasing availability of data and computing resources makes non-traditional algorithms, such as machine learning algorithms, a particular technological opportunity in the healthcare system. However, incorporating such non-traditional algorithms into any medical technology presents many challenges.

[0005] Some surgical systems may include a centralized surgical computing device that may interact with multiple surgical devices. It may be desirable to have a surgical system configured in a manner that may avoid or partially avoid the use of a centralized computing device. Summary of the Invention [Means for solving the problem]

[0006] According to one embodiment of the present invention, a first surgical device configured for peer monitoring is provided, the first surgical device comprising a processor configured to: determine that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establish a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record a surgical task performed by the second surgical device; and monitor or record surgical information associated with the performance of the surgical task by the second surgical device using the established peer-to-peer connection.

[0007] The present invention may enable a central computing system (e.g., a hub or edge device) to offload less critical computations to a peer-to-peer network, which may have the advantage of reducing dependency on the central processing system.

[0008] The term "device" includes surgical instruments (e.g., used to manipulate tissue) as well as computing devices (e.g., hub or edge devices).

[0009] In one embodiment, the first surgical device and / or the second surgical device are located in an operating room, optionally within a sterile field. The hub may be located within the operating room, but is typically located outside the sterile field. For example, the first surgical device and / or the second surgical device may be an instrument configured to be held by a surgeon or a robot to perform a task (directly on a patient) in a surgical procedure. The first surgical device and / or the second surgical device may be, but are not limited to, a surgical stapler, an energy device, an electrosurgical instrument, an ultrasonic instrument, a combination electrosurgical / ultrasonic instrument, and / or a combination stapler / electrosurgical device.

[0010] Alternatively, the second surgical device may be a computing device such as a hub or edge device, in which case the first surgical device monitors the hub / edge computing device by, for example, monitoring data received by the hub from a third surgical device (such as a surgical stapler, an energy device, an electrosurgical instrument, an ultrasonic instrument, a combination electrosurgical / ultrasonic instrument, and / or a combination stapler / electrosurgical device) and monitoring or recording data transmitted from the hub to the third surgical device in the performance of a surgical task (which may include, for example, control settings and / or recommendations / warnings for the third surgical device).

[0011] In one embodiment, the first surgical device is a stapler and the second surgical device is an energy device.

[0012] "Monitoring or recording" may include receiving and storing (at the first surgical device) data from the second surgical device or accessing data stored within the second surgical device.

[0013] The first surgical device may request, from the second surgical device, information associated with the second surgical device performing the surgical task. Alternatively, the second surgical device may push information to the first surgical device. The information may be pushed periodically or aperiodically.

[0014] In one embodiment, a first surgical device establishes a peer-to-peer connection with a plurality of surgical devices.

[0015] There may be an exchange of data between the first surgical device and the second surgical device.

[0016] The information may be stored in a memory of the first surgical device, and the information may be extracted to analyze the performance of a task by a second surgical device and / or the interaction (e.g., data exchange) between the first and second surgical devices. The first surgical device may be configured to output the information to a third surgical device, which may include a central computing device (e.g., a hub / edge device) in an operating room or hospital, or to a remote server or the cloud.

[0017] The processor may be further configured to obtain information regarding the second surgical device directly from the second surgical device.

[0018] By receiving information directly from the second surgical device, the use of a central processing system to relay data between devices is avoided.

[0019] In one embodiment, a first surgical device establishes a peer-to-peer connection with a plurality of surgical devices, one or more of which may transmit device information directly to the first surgical device and one or more of which may transmit device information to the first surgical device via central computing (e.g., a hub / edge device).

[0020] The first surgical device may be configured to process information from the second surgical device and provide feedback to the second surgical device or another surgical device (which may be a hub / edge device).

[0021] The feedback may include at least one of a recommendation to change a control setting, a recommendation to replace a replaceable component, or a warning.

[0022] The processor may be further configured to process information associated with the second surgical device to generate suggested changes to control settings of the second surgical device, adjust the control settings of the second surgical device, or send a recommendation for the change in control settings to the second surgical device.

[0023] The first surgical device may monitor or record surgical information associated with the second surgical device to oversee tasks performed by the second surgical device. For example, the first surgical device may process information from the second surgical device to enhance surgical control of the second surgical device and / or provide recommendations or warnings to the user via the second surgical device (or a display other than the second surgical device).

[0024] In one example, if the second surgical device is a surgical stapler, the first surgical device may receive information related to the tissue thickness of the tissue captured between its jaws. The first surgical device may then provide recommendations to the first surgical device regarding starting motor speed, clamping force, firing force, etc., and during the surgical procedure, the first surgical device may receive motor speed information from the stapler that may indicate a higher than expected motor speed. Based on the type of tissue being operated on (information received by the first surgical device, which may be information received from an operator input, from a peer surgical device, or from a hub / edge device, for example), the first surgical device may recommend that the second surgical device increase or decrease the motor speed (depending on the tissue type and tissue thickness) to avoid tearing the tissue or ending up with an incomplete staple line.

[0025] In another example, the second instrument is an energy device that provides tissue incision. The information received by the first surgical device may include parameters including power, the time energy is delivered to the tissue, tissue impedance, etc. The first surgical device may process this information to identify an optimal location for the tissue incision (e.g., based on the nature of the tissue or the disease state of the tissue, areas that avoid blood vessels, etc.). Additionally, or alternatively, the first surgical device may determine from processing the data that coagulation is not complete and therefore provide feedback to the second surgical device to increase the power level, change the frequency of the delivered energy, increase the time of energy application, etc.

[0026] Upon processing the information, the first surgical device may establish that the tissue is of a particular thickness and that the thickness is better suited to a different staple cartridge than the cartridge currently attached to the stapler. Accordingly, the first surgical device may transmit a recommendation to replace the current cartridge with a more suitable cartridge, which may be displayed on a display of the second surgical device (or on a display elsewhere).

[0027] The first surgical device may process the information and determine that a parameter is outside of an expected or safe range, for example, the tissue thickness is too thick for the cartridge currently attached to the surgical stapler, the firing force value is outside of an expected range and there may be a risk of tissue tearing, the energy device is insufficiently coagulating the tissue, etc. The first surgical device may send this feedback to the second surgical device as an instruction to display a warning message on a display of the second surgical device (or send it to be displayed elsewhere), thereby alerting the user to the potential risk.

[0028] The second surgical device may be configured to receive feedback and is configured to at least one of change a control setting, display a recommendation to change a replaceable component, or display a warning message.

[0029] The processor may be further configured to process information associated with the second surgical device to generate suggested changes to the control settings of the first surgical device and adjust the control settings of the first surgical device based on the suggested changes.

[0030] The first surgical device may monitor or record surgical information associated with the second surgical device in order to adjust its own performance of the surgical task based on the information collected from the second surgical device.

[0031] For example, if the first surgical device is a stapler and the second surgical device is an energy device, information about the area incised by the energy device and information about the thickness of the tissue may be used to provide operating parameter, control setting, or cartridge type recommendations for the stapler that will be used to perform the stapling task after the energy incision task.

[0032] The processor configured to determine that the first surgical device is capable of establishing a peer-to-peer connection with the second surgical device may further comprise a processor configured to send an indication of a discovery request to the second surgical device and receive an indication of a response message from the second surgical device, or receive an indication of a discovery request from the second surgical device and send an indication of a response message to the second surgical device.

[0033] In one embodiment, discovery of devices to establish peer-to-peer connections is implemented by the devices themselves, for example, via direct messaging between devices.

[0034] In an alternative example, a central computing device, e.g., a hub or edge device, may pair with both the first and second surgical devices, determine what roles should be assigned to the first and second surgical devices (and then provide data regarding the assigned roles to the first and second surgical devices).

[0035] The discovery operation may be performed by the first surgical device or by the surgical hub / edge device at the beginning of a surgical procedure or during a transition phase from one surgical step to a subsequent surgical step of a surgical procedure. As surgical steps / tasks change, the devices used, the data collected, and the processing required for surgical device control may change, so it may be advantageous to change the peer-to-peer connection during the transition phase between surgical steps.

[0036] The response message instructions may include information about the type and capabilities of the surgical device.

[0037] The type of surgical device and the capabilities of the surgical device may be stored in memory of the control circuitry of the surgical device.

[0038] Device type and capability information may be used to establish whether peer-to-peer connections are possible and which devices should and should not be monitored.

[0039] To be able to establish a peer-to-peer connection, the first surgical device must determine whether it has established connections with devices that it needs to monitor during the procedure (which may be determined by the treatment plan or the list of devices used in the procedure) and whether it has adequate processing power to handle the monitoring / recording of data while the second surgical device is performing the surgical task.

[0040] The first surgical device may receive a list of surgical tasks (or steps) of the surgical procedure and the types of surgical devices to be used to perform each of those surgical tasks. For example, the first surgical device may receive data that one of the tasks is a cutting and stapling / sealing task and that the device type for that task is a surgical cutter stapler or an energy device. The list of types of surgical devices to be used to perform each of the surgical tasks may include information on the required performance characteristics of the type of surgical device, such as the closure gap between the jaws for a surgical stapler, firing force, clamping force, etc., or the operating frequency of an energy device for a particular device that may be used to perform the surgical task.

[0041] The first surgical device can then determine whether the response message it receives pertains to a device that can perform the task in the procedure and whether it can monitor it based, for example, on the data it processes (e.g., the size of the data it receives, whether it has a suitable model to process or whether it can acquire a suitable model to properly process that data, etc.) Additionally, the ability of the second surgical device to perform the surgical task may be based on an operating range (e.g., operating frequency or power, FTF, FTC, etc.).

[0042] The first surgical device may receive this information by interrogating memory or from a processor external to the device (e.g., from a hub / edge device). The memory may contain task and device data for multiple different surgical procedures, and the first surgical device may receive (e.g., from a hub) instructions for the surgical procedure that is about to be performed or is currently being performed (and, e.g., the current task of that procedure being performed). The device and / or hub may be able to determine the surgical procedure or the current task in the surgical procedure using situational awareness (e.g., based on data about the surgical procedures and their steps / tasks, a list of devices used in those steps, information about which devices are active, and / or data streams from those devices).

[0043] In one example, a first surgical device may receive (e.g., from a surgical hub / edge device) a list of potential surgical devices that it may monitor. The monitoring surgical device may also receive instructions identifying surgical devices that the monitoring surgical device may be able to monitor directly and surgical devices that the monitoring surgical device may be able to monitor in cooperation with the surgical hub / edge device.

[0044] In one example, a first surgical device may receive instructions from a central computing device to monitor a plurality of peer surgical devices. The instructions may include a list of identification tags associated with the peer surgical devices. The monitoring surgical device may store the list of peer surgical devices to be monitored.

[0045] Determining that the first surgical device is capable of establishing a peer-to-peer connection with the second surgical device may include determining that one of the first surgical device and the second surgical device is capable of accessing or receiving data from the other.

[0046] The device may also be required to be able to process the data, for example, have the processing power to use the model to process the data and provide feedback or recommendations to the other device.

[0047] The ability of the first surgical device to be a monitoring surgical device may include the first surgical device having the ability to remotely access surgical information at a second surgical device.

[0048] The determination of which of the first and second surgical devices will be the monitoring device and which will be the monitored device may be based on the capabilities of the first and second surgical devices.

[0049] The ability of the first surgical device to be a monitoring surgical device may include having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device.

[0050] The first surgical device having the capability to be a monitoring surgical device and the second surgical device having the capability to be monitored by the first surgical device may be determined based on negotiation between the first surgical device and the second surgical device.

[0051] The negotiation may include the transfer of data between the two devices and the application of one or more rules to determine role assignment, e.g., which device will monitor and which device will be monitored. The determination may depend on the speed or capabilities of each of the devices, the memory capacity of the devices, timing (e.g., which device sent the discovery request), the strength of connectivity between the devices, etc. The determination may also be based on whether the type of surgical device will be used for the surgical task of the surgical procedure and, optionally, the capabilities of the type of surgical device required for that task or the capabilities of the monitoring instrument (e.g., higher processing speed for processing data, newer model for processing data, larger memory, etc.).

[0052] When both the first surgical device and the second surgical device have the capability to be a monitoring device and a monitored device and both are used in tasks in a surgical procedure, the determination of which is the monitoring device can be based on timing, such as which device first sends or receives a discovery / response message.

[0053] The second surgical device may be a surgical hub.

[0054] The processor may be further configured to establish a second peer-to-peer connection with a third surgical device, initiate monitoring and recording of the second surgical device using the established peer-to-peer connection with the second surgical device, and initiate monitoring and recording of the third surgical device using the established second peer-to-peer connection.

[0055] The first surgical device may be used to monitor a third surgical device through its connection to the second surgical device, for example, because the second and third surgical devices are connected via an additional peer-to-peer connection. The first surgical device may request access to stored data of the second surgical device, which stored data may include data related to the third surgical device, or may request that the second surgical device transmit data on the third surgical device to it. In this configuration, for example, the second surgical device may act as a relay between the first and third surgical devices. This may be the case, for example, when a direct connection between the first and third surgical devices may not be possible or may be inefficient.

[0056] By monitoring through the second surgical device, the first surgical device can obtain information regarding the operation of the third device without establishing a direct connection with the third device (which may be considered eavesdropping on the operation of the third device).

[0057] This connected peer-to-peer network allows the first surgical device to monitor devices in addition to the second surgical device, thereby gathering more information about the current procedure and device operation (which can be used to control or enhance control of one or more of the first, second, or third surgical devices).

[0058] Monitoring and recording a second surgical device using an established peer-to-peer connection with the second surgical device and monitoring and recording a third surgical device using the established second peer-to-peer connection may be performed simultaneously.

[0059] According to a further embodiment of the present invention, there is provided a first surgical device configured to be peer monitored, the first surgical device comprising a processor configured to: determine that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establish a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the second surgical device to monitor and record a surgical task performed by the first surgical device; and transmit surgical information associated with the performance of the surgical task to the second surgical device using the established peer-to-peer connection.

[0060] According to a further embodiment of the present invention, there is provided a system configured for peer monitoring, the system comprising a hub or edge device having a processor configured to establish a connection with at least a first surgical device and a second surgical device, assign a monitoring role to the first surgical device based on one or more of the types of the first and second surgical devices, a task performed in the surgical procedure, and a capability of the first and second surgical devices, and assign a monitored role to the second surgical device based on at least one of the types of the first and second surgical devices, the task performed in the surgical procedure, and a capability of the first and second surgical devices, the first surgical device configured to establish a peer-to-peer connection with the second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device, and the first surgical device configured to monitor or record surgical information associated with the performance of surgical tasks by the second surgical device using the established peer-to-peer connection.

[0061] According to a further embodiment of the present invention, there is provided a peer-to-peer surgical information monitoring method implemented on a first surgical device, the method including: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks on the second surgical device; and monitoring or recording surgical information associated with the second surgical device using the established peer-to-peer connection with the second surgical device.

[0062] Information regarding the second surgical device may be obtained directly from the second surgical device.

[0063] The method may further include processing information associated with the second surgical device to generate suggested changes to control settings of the second surgical device, and adjusting the control settings of the second surgical device or transmitting a recommendation for the change in control settings to the second surgical device.

[0064] The method may further include processing information associated with the second surgical device to generate proposed changes to control settings of the first surgical device, and adjusting the control settings of the first surgical device based on the proposed changes.

[0065] Determining that the first surgical device is capable of establishing a peer-to-peer connection with the second surgical device may further include sending an indication of a discovery request to the second surgical device and receiving an indication of a response message from the second surgical device, or receiving an indication of a discovery request from the second surgical device and sending an indication of a response message to the second surgical device.

[0066] The response message instructions may include information about the type and capabilities of the surgical device.

[0067] Determining that the first surgical device is capable of establishing a peer-to-peer connection with the second surgical device may include determining that one of the first surgical device and the second surgical device is capable of accessing or receiving data from the other.

[0068] The determination of which of the first and second surgical devices will be the monitoring device and which will be the monitored device may be based on the capabilities of the first and second surgical devices.

[0069] The ability of the first surgical device to be a monitoring surgical device may include having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device.

[0070] The first surgical device having the capability to be a monitoring surgical device and the second surgical device having the capability to be monitored by the first surgical device may be determined based on negotiation between the first surgical device and the second surgical device.

[0071] The second surgical device may be a surgical hub.

[0072] The method may further include establishing a second peer-to-peer connection with a third surgical device, monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device, and monitoring and recording the third surgical device using the established second peer-to-peer connection.

[0073] Monitoring and recording a second surgical device using an established peer-to-peer connection with the second surgical device and monitoring and recording a third surgical device using the established second peer-to-peer connection may be performed simultaneously.

[0074] According to further embodiments of the present invention, a computing program is provided that, when executed by a processor on a first surgical device, causes the processor to perform any of the methods described above. Systems, methods, and means may be provided for a smart surgical instrument or surgical device to monitor other surgical instruments or surgical devices in a peer-to-peer interconnected surgical ecosystem. The monitoring and / or recording may be performed by a surgical device that may be configured as a monitoring surgical device. The monitoring surgical device may use the peer-to-peer surgical ecosystem to monitor and / or record surgical information associated with surgical tasks on peer surgical instruments, for example, without a central surgical hub.

[0075] In one example, a surgical device may determine that it has the capability to monitor and record a surgical task associated with a surgical procedure being performed on a second peer surgical device. The surgical device may determine whether it has the capability to be configured as a monitoring surgical device. The capability of a surgical device to be a monitoring surgical device may include the surgical device having the capability to access surgical information associated with the surgical task on the second peer surgical device and / or having the capability to set (e.g., remotely set) surgical parameters associated with the surgical task or surgical parameters on or of the second peer surgical device based at least on surgical information accessed and / or received from the second peer surgical device. The accessed surgical information may include surgical data associated with the patient, the medical professional, the surgical task, and / or the second peer surgical device. Based on the determination, the surgical device can configure itself as a monitoring surgical device and may configure itself as a monitoring surgical device. In one example, the role of the first surgical device as the monitoring surgical device and the role of the second surgical device as the peer surgical device may be based on a negotiation between the monitoring surgical device and the second peer surgical device.

[0076] The monitoring surgical device may determine that the second surgical device has the capability to be a peer surgical device that may be monitored by the monitoring surgical device. The capability to be a peer surgical device may include having the capability to establish a peer-to-peer connection with the monitoring surgical device and / or having the capability to collect surgical data associated with a patient, medical professional, or surgical device and report the collected surgical data to the monitoring surgical device. The surgical device may configure the second surgical device as a peer surgical device.

[0077] A surgical device may establish a peer-to-peer connection with a second surgical device. A peer-to-peer connection may be established between a first surgical device and a second surgical device such that the first surgical device monitors and records information associated with a surgical task on the second surgical device. The peer-to-peer connection between the two surgical devices may be established using a wired interface (e.g., via a local area network (LAN)), a wireless interface (e.g., a WiFi interface (WiFi 6, WiFi 6E, etc.), a Bluetooth X interface, etc.), and / or an optical interface (e.g., an optical fiber-based LAN). The two surgical devices may exchange surgical information and other parameters without a central surgical computing device (e.g., a surgical hub).

[0078] After a peer-to-peer connection is established between the two surgical devices, the monitoring surgical device may begin monitoring and recording surgical information and exchanging surgical information with each other. In one example, the monitoring surgical device may use the established peer-to-peer connection to monitor and / or record surgical information associated with a surgical task on the peer surgical device. In one example, the peer surgical device may use the established peer-to-peer connection to periodically or aperiodically (e.g., based on a trigger) send / report surgical information associated with a surgical task to the monitoring surgical device. [Brief explanation of the drawings]

[0079] [Figure 1] FIG. 1 is a block diagram of a computer-implemented surgical system. [Figure 2] 1 illustrates an exemplary surgical system in an operating room. [Figure 3] 1 illustrates exemplary surgical hubs paired with various systems. [Figure 4] Illustrates a surgical data network having a set of communicating surgical hubs configured to connect with a set of sensing systems, an environmental sensing system, a set of devices, and the like. [Figure 5] 1 illustrates a logic diagram of a control system for a surgical instrument or device. [Figure 6] 1 illustrates an exemplary surgical system including a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter. [Figure 7A] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 7B] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 7C] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 7D] 1A-1C show, respectively, an example surgical system information matrix, an example information flow in a surgical system, an example information flow in a surgical system with a surgical robot, and a diagram of surgical information in the context of a procedure. [Figure 8A]1A and 1B show an exemplary supervised learning framework and an exemplary unsupervised learning framework, respectively. [Figure 8B] 1A and 1B show an exemplary supervised learning framework and an exemplary unsupervised learning framework, respectively. [Figure 9] 1 illustrates an example overview of data flow within a peer-to-peer interconnected surgical system. [Figure 10] 1 illustrates an example sequence for interconnecting a surgical hub / edge device with a surgical device. [Figure 11] a shows an exemplary sequence for interconnecting a surgical hub / edge device and a surgical device. [Figure 12] 1 illustrates an example of a relationship between a surgical hub / edge device and a surgical device. [Figure 13] For example, an example of a peer-to-peer interconnected surgical system device utilized for remote monitoring / recording without the use of a central surgical hub is shown. [Figure 14] 1 illustrates a discovery mechanism used to assign roles (eg, monitor role and / or peer role) to surgical devices that may be utilized in a surgical procedure. DETAILED DESCRIPTION OF THE INVENTION

[0080] 1 is a block diagram of a computer-implemented surgical system 100. An exemplary surgical system, such as surgical system 100, may include one or more surgical systems (e.g., surgical subsystems) 102, 103, 104. For example, surgical system 102 may include a computer-implemented bidirectional surgical system. For example, surgical systems 102, 103, 104 may include a surgical computing system, such as a surgical hub 106 and / or a computing device 116, in communication with a cloud computing system 108. Cloud computing system 108 may include a cloud server 109 and a cloud storage unit 110.

[0081] The surgical systems 102, 103, 104 may each be computer-enabled surgical instruments and devices. For example, the surgical systems 102, 103, 104 may include a wearable sensing system 111, a human interface system 112, a robotic system 113, one or more intelligent instruments 114, an environmental sensing system 115, and / or others. The wearable sensing system 111 may include one or more devices used to sense aspects of the status and activity of individuals within the surgical environment. For example, the wearable sensing system 111 may include a healthcare provider sensing system and / or a patient sensing system.

[0082] The human interface system 112 may include devices that allow individuals to interact with the surgical systems 102, 103, 104 and / or the cloud computing system 108. The human interface system 112 may include human interface devices.

[0083] The robotic system 113 may include a surgical robotic device, such as a surgical robot. The robotic system 113 may enable robotic surgical procedures. The robotic system 113 may receive information, settings, programming, control, etc. from the surgical hub 106; for example, the robotic system 113 may transmit data, such as sensor data, feedback information, video information, and operation logs, to the surgical hub 106.

[0084] Environmental sensing system 115 may include, for example, one or more devices used to measure one or more environmental attributes, as further described in Figure 2. Robotic system 113 may include multiple devices used to perform a surgical procedure, as further described in Figure 2.

[0085] The surgical system 102 may communicate with a remote server 109, which may be part of a cloud computing system 108. In one example, the surgical system 102 may communicate with the remote server 109 via a networked connection, such as an Internet connection (e.g., business internet service, T3, cable / FIOS networking node, etc.). The surgical system 102 and / or its components may communicate with the remote server 109 via a cellular transmission / reception point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), long term evolution (LTE) or 4G, LTE-Advanced (LTE-A), new radio (NR), or 5G.

[0086] In one example, the surgical hub 106 may facilitate displaying images from a surgical imaging device, such as a laparoscope. The surgical hub 106 may have collaborative interaction with other local systems to facilitate displaying information related to those local systems. The surgical hub 106 may interact with one or more sensing systems 111, 115, one or more intelligent instruments 114, and / or multiple displays. For example, the surgical hub 106 may be configured to collect measurement data from one or more sensing systems 111, 115 and send notification or control messages to one or more sensing systems 111, 115. The surgical hub 106 may send and / or receive information, including notification information, to and / or from a human interface system 112. The human interface system 112 may include one or more human interface devices (HIDs). The surgical hub 106 may send and / or receive notification or control information for audio, display, and / or control information to various devices in communication with the surgical hub.

[0087] For example, the sensing systems 111, 115 may include a wearable sensing system 111 (which may include one or more HCP sensing systems and one or more patient sensing systems) and an environmental sensing system 115. The one or more sensing systems 111, 115 may measure data related to various biomarkers. The one or more sensing systems 111, 115 may measure the biomarkers using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The one or more sensors may measure the biomarkers as described herein using one of many of the following sensing technologies: photoplethysmography, electrocardiography, electroencephalography, colorimetric, impedimentary, potentiometric, amperometric, etc.

[0088] Biomarkers measured by one or more sensing systems 111, 115 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood potential of hydrogen, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, coughing and sneezing, gastrointestinal motility, gastrointestinal imaging, airway bacteria, edema, mental status, sweat, circulating tumor cells, autonomic tone, circadian rhythm, and / or menstrual cycle.

[0089] The biomarkers may relate to physiological systems, which may include, but are not limited to, behavioral and psychological, cardiovascular, renal, dermatological, nervous, gastrointestinal, respiratory, endocrine, immune, oncological, musculoskeletal, and / or reproductive systems. Information from the biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgical system 100. Information from the biomarkers may be determined and / or used, for example, by the computer-implemented patient and surgical system 100 to improve the system and / or improve patient outcomes. The one or more sensing systems 111, 115, biomarkers, and physiological systems are described in more detail in U.S. patent application Ser. No. 17 / 156,287 (Attorney Docket No. END9290USNP1), entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS," filed on January 22, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0090] FIG. 2 shows an example of a surgical system 202 in a surgical operating room. As illustrated in FIG. 2, a patient is being operated on by one or more health care professionals (HCPs). The HCPs are monitored by one or more HCP sensing systems 220 worn by the HCPs. The HCPs and the environment surrounding the HCPs may also be monitored by one or more environmental sensing systems including, for example, a set of cameras 221, a set of microphones 222, and other sensors that may be deployed in the operating room. The HCP sensing systems 220 and the environmental sensing systems may communicate with a surgical hub 206, which may then communicate with one or more cloud servers 209 of a cloud computing system 208, as shown in FIG. 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as HCP position within the operating room, HCP movement, ambient noise within the operating room, temperature / humidity within the operating room, etc.

[0091] As illustrated in FIG. 2 , a primary display 223 and one or more audio output devices (e.g., speaker 219) are positioned in the sterile field for visibility to the operator at the operating table 224. Additionally, a visualization / notification tower 226 is positioned outside the sterile field. The visualization / notification tower 226 may include a first non-sterile human interactive device (HID) 227 and a second non-sterile HID 229 facing opposite each other. The HIDs may be displays or displays with touchscreens that allow a human to directly interface with the HIDs. A human interface system guided by the surgical hub 206 may be configured to utilize the HIDs 227, 229, and 223 to coordinate information flow to operators inside and outside the sterile field. In one example, the surgical hub 206 may cause an HID (e.g., primary HID 223) to display notifications and / or information regarding the patient and / or surgical procedure steps. In one example, the surgical hub 206 may prompt and / or receive input from a person in the sterile field or non-sterile area. In one example, the surgical hub 206 may cause the HID to display a snapshot of the surgical site as recorded by the imaging device 230 on the non-sterile HID 227 or 229 while maintaining a live view of the surgical site on the main HID 223. The snapshot on the non-sterile display 227 or 229 may, for example, allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0092] In one aspect, the surgical hub 206 may be configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 226 to a primary display 223 in the sterile field for viewing by a sterile operator at the operating table. In one example, the input may be in the form of a correction to a snapshot displayed on the non-sterile display 227 or 229, which may be sent by the surgical hub 206 to the primary display 223.

[0093] 2, a surgical instrument 231 is used as part of a surgical system 202 in a surgical procedure. A hub 206 may be configured to coordinate information flow to the display of the surgical instrument 231, as described, for example, in U.S. Patent Application Publication No. 2019-0200844(A1), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 226 can be sent by the hub 206 to a surgical instrument display in the sterile field, where it can be viewed by the operator of the surgical instrument 231. Exemplary surgical instruments suitable for use with surgical system 202 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety.

[0094] FIG. 2 illustrates an example of a surgical system 202 being used to perform a surgical procedure on a patient lying on an operating table 224 in an operating room 235. A robotic system 234 may be used as part of the surgical system 202 in the surgical procedure. The robotic system 234 may include a surgeon's console 236, a patient side cart 232 (surgical robot), and a surgical robot hub 233. The patient side cart 232 can manipulate at least one detachably coupled surgical tool 237 through a minimally invasive incision within the patient's body while the surgeon views the surgical site through the surgeon's console 236. Images of the surgical site can be acquired by a medical imaging device 230, which can be manipulated by the patient side cart 232 to orient the imaging device 230. The robotic hub 233 can be used to process the images of the surgical site and then display them to the surgeon through the surgeon's console 236.

[0095] Other types of robotic systems can be readily adapted for use with surgical system 202. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0096] Various examples of cloud-based analytics methods that may be performed by cloud computing system 208 and that are suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0097] In various embodiments, the imaging device 230 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0098] The optical components of the imaging device 230 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0099] The one or more illumination sources may be configured to emit electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and is sometimes referred to as visible light or simply light. The human eye typically responds to wavelengths in air between about 380 nm and about 750 nm.

[0100] The invisible spectrum (e.g., non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0101] In various aspects, imaging device 230 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0102] The imaging device may use multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging allows for the extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after a surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical equipment is necessary during any surgical procedure. The strict hygienic and sterilization conditions required in an "operating room," i.e., an operating room or procedure room, require the highest possible sterility of all medical devices and equipment. Part of the sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 230 and its accessories and components. It will be understood that the sterile field may be considered a specific area, such as in a tray or on a sterile towel, that is deemed free of microorganisms, or the sterile field may be considered the area immediately surrounding the patient being prepared for the surgical procedure. The sterile field may include properly clothed and hand-washed team members, as well as all equipment and fixtures in the area.

[0103] The wearable sensing system 211 illustrated in FIG. 1 may include one or more sensing systems, such as an HCP sensing system 220 as shown in FIG. 2. The HCP sensing system 220 may include a sensing system that monitors and detects a set of physical and / or physiological conditions of a healthcare professional (HCP). An HCP may generally be one or more healthcare professionals assisting a surgeon or other healthcare provider. In one example, the sensing system 220 may measure a set of biomarkers to monitor the HCP's heart rate. In one example, the sensing system 220 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movement and / or shaking and determine tremor magnitude and frequency. The sensing system 220 may transmit measurement data associated with the set of biomarkers and data associated with the surgeon's physical condition to the surgical hub 206 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 206. For example, the environmental sensing devices may include a camera 221 for detecting the HCP's hand / body position. The environmental sensing devices may include a microphone 222 for measuring ambient noise within the operating room. Other environmental sensing devices may include devices such as a thermometer for measuring temperature and a hygrometer for measuring ambient humidity within the operating room. The surgical hub 206, alone or in communication with a cloud computing system, may use the surgeon biomarker measurement data and / or environmental sensing information to modify the control algorithms of handheld instruments or the average latency of a robotic interface, for example, to minimize tremor. In one example, the HCP sensing system 220 may measure one or more surgeon biomarkers associated with the HCP and transmit measurement data associated with the surgeon biomarkers to the surgical hub 206.The HCP sensing system 220 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), Wi-Fi, etc. Surgeon biomarkers may include one or more of stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels associated with the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention level, etc.

[0104] The surgical hub 206 may use the surgeon biomarker measurement data associated with the HCP to adaptively control one or more surgical instruments 231. For example, the surgical hub 206 may send a control program to the surgical instrument 231 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 206 may send the control program based on situational awareness and / or context regarding the importance or criticality of the task. The control program may instruct the instrument to change operation to provide more control when control is needed.

[0105] FIG. 3 illustrates an exemplary surgical system 302 having a surgical hub 306. The surgical hub 306 may be paired with a wearable sensing system 311, an environmental sensing system 315, a human interface system 312, a robotic system 313, and an intelligent instrument 314 via a modular control. The hub 306 includes a display 348, an imaging module 349, a generator module 350, a communications module 356, a processor module 357, a storage array 358, and an operating room mapping module 359. In certain embodiments, as illustrated in FIG. 3, the hub 306 further includes a smoke evacuation module 354 and / or a suction / irrigation module 355. The various modules and systems may be connected to the modular control either directly via a router or via the communications module 356. Operating room devices may be coupled to cloud computing resources and data storage via the modular control. The human interface system 312 may include a display subsystem and a notification subsystem.

[0106] The modular controller may be coupled to a non-contact sensor module. The non-contact sensor module may use ultrasonic, laser-type, and / or similar non-contact measurement devices to measure the dimensions of the operating room and generate a map of the operating room. Other distance sensors may be used to determine the boundaries of the operating room. As described in U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," which is incorporated herein by reference in its entirety, under the heading "Surgical Hub Spatial Awareness Within an Operating Room," an ultrasound-based non-contact sensor module may scan the operating room by transmitting bursts of ultrasound and receiving echoes as the bursts reflect off the exterior walls of the operating room. The sensor module may be configured to determine the size of the operating room and adjust Bluetooth pairing distance limits. For example, a laser-based non-contact sensor module can scan an operating room by transmitting laser light pulses, receiving laser light pulses that reflect off the exterior walls of the operating room, and comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust Bluetooth pairing distance limits.

[0107] During surgical procedures, the application of energy to tissue for sealing and / or cutting is generally associated with smoke evacuation, aspiration of excess fluid, and / or irrigation of tissue. Fluid, power, and / or data lines from different sources often become tangled during surgical procedures. Addressing this issue can result in valuable time being lost during a surgical procedure. Untangling the lines may require unplugging them from their corresponding modules, which may require resetting the modules. The hub modular enclosure 360 ​​reduces the frequency of such line tangles by providing a unified environment for managing power, data, and fluid lines. Aspects of the present disclosure present a surgical hub 306 for use in surgical procedures involving the application of energy to tissue at a surgical site.

[0108] The surgical hub 306 includes a hub enclosure 360 ​​and a combination generator module slidably received within a docking station of the hub enclosure 360. The docking station includes data and power contacts. The combination generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit. In one aspect, the combination generator module also includes a smoke evacuation component, at least one energy delivery cable for connecting the combination generator module to a surgical instrument, at least one smoke evacuation component configured to evacuate smoke, fluid, and / or particulates generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the smoke evacuation component. In one aspect, the fluid line may be a first fluid line and a second fluid line may extend from the remote surgical site to the suction and irrigation module 355, which is slidably received within the hub enclosure 360. In one aspect, the hub enclosure 360 ​​may include a fluid interface.

[0109] Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another, different energy type may be more beneficial for sealing tissue. For example, a bipolar generator may be used to seal tissue, while an ultrasonic generator may be used to cut the sealed tissue. Aspects of the present disclosure present a solution in which a hub modular enclosure 360 ​​is configured to house different generators and facilitate bidirectional communication between them. The hub modular enclosure 360 ​​may allow for quick removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue and a first docking station including a first docking port including first data contacts and first power contacts, wherein the first energy generator module is slidably movable into electrical engagement with the power contacts and the data contacts, and the first energy generator module is slidably movable out of electrical engagement with the first power contacts and the first data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy different from the first energy for application to tissue and a second docking station including a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power contacts and the data contacts, and the second energy generator module is slidably movable out of electrical engagement with the second power contacts and the second data contacts. In addition, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port configured to facilitate communication between the first energy generator module and the second energy generator module.Referring to FIG. 3 , an aspect of the disclosure is presented regarding a hub modular enclosure 360 ​​that allows for modular integration of a generator module 350, a smoke evacuation module 354, and a suction / irrigation module 355. The hub modular enclosure 360 ​​further facilitates bidirectional communication between the modules 359, 354, and 355. The generator module 350 can comprise integrated monopolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably insertable into the hub modular enclosure 360. The generator module 350 can be configured to connect to a monopolar device 351, a bipolar device 352, and an ultrasonic device 353. Alternatively, the generator module 350 may comprise a series of monopolar, bipolar, and / or ultrasonic generator modules that interact through the hub modular enclosure 360. The hub modular enclosure 360 ​​may be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub modular enclosure 360 ​​so that the multiple generators act as a single generator.

[0110] FIG. 4 illustrates a surgical data network having a set of communication hubs configured to connect a set of sensing systems, environmental sensing systems, and other modular devices located in one or more operating rooms, patient recovery rooms, or rooms within a medical facility specially equipped for surgery to a cloud in accordance with at least one embodiment of the present disclosure.

[0111] 4, surgical hub system 460 may include a modular communications hub 465 configured to connect modular devices located within a medical facility to a cloud-based system (e.g., a cloud computing system 464 that may include a remote server 467 coupled to remote storage 468). The modular communications hub 465 and devices may be connected in a room within the medical facility specially equipped for surgical procedures. In one aspect, modular communications hub 465 may include a network hub 461 and / or a network switch 462 in communication with a network router 466. Modular communications hub 465 may be coupled to a local computer system 463 to provide local computer processing and data manipulation.

[0112] The computer system 463 may include a processor and a network interface 20100. The processor may be coupled to a communication module, storage, memory, non-volatile memory, and input / output (I / O) interfaces via a system bus. The system bus can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures, including, but not limited to, a 9-bit bus, an Industrial Standard Architecture (ISA), a Micro-Channel Architecture (MSA), an Extended ISA (EISA), an Intelligent Drive Electronics (IDE), a VESA Local Bus (VLB), a Peripheral Component Interconnect (PCI), a USB, an Advanced Graphics Port (AGP), a Personal Computer Memory Card International Association (PCMCIA), a Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0113] The processor may be any single-core or multi-core processor, such as that known by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which may include on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle serial random access memory (SRAM), internal read-only memory (ROM) loaded with StellarisWare® software, 2 KB of electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more Quadrature Encoder Input (QEI) analog, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. The NI PXIe-4230 includes a 100-MHz SPI, ...

[0114] In one example, the processor may include a safety controller, which includes two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller may be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0115] It should be understood that the computer system 463, in a suitable operating environment, may include software that acts as an intermediary between the described users and the basic computer resources. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate resources of the computer system. System applications may take advantage of resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0116] A user may input commands or information into the computer system 463 through input devices coupled to the I / O interface. Input devices may include, but are not limited to, pointing devices such as a mouse, trackball, stylus, or touchpad; keyboards; microphones; joysticks; gamepads; satellite dishes; scanners; TV tuner cards; digital cameras; digital video cameras; webcams; and the like. These and other input devices connect to the processor through the system bus via interface ports. Interface ports include, for example, serial ports, parallel ports, game ports, and USB. Output devices use some of the same types of ports as input devices. Thus, for example, a USB port may be used to provide input to the computer system 463 and to output information from the computer system 463 to an output device. Output adapters may be provided to illustrate that there may be several output devices, such as monitors, displays, speakers, and printers, among other output devices that may require special adapters. Output adapters may include, by way of example and not limitation, video and sound cards that provide a means of connection between the output device and the system bus. It should be noted that other devices and / or systems of devices, such as remote computers, may provide both input and output capabilities.

[0117] The computer system 463 may operate in a networked environment using logical connections to one or more remote computers, such as a cloud computer, or a local computer. The remote cloud computer may be a personal computer, a server, a router, a network PC, a workstation, a microprocessor-based appliance, a peer device, or other common network node, but typically includes many or all of the elements described with respect to a computer system. For simplicity, only memory storage devices are illustrated along with the remote computer. The remote computer may be logically connected to the computer system through a network interface and subsequently physically connected via a communication connection. The network interface may encompass communication networks such as a local area network (LAN) and a wide area network (WAN). LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, etc. WAN technologies may include, but are not limited to, point-to-point links, circuit-switched networks such as Integrated Services Digital Networks (ISDN) and its variants, packet-switched networks, and Digital Subscriber Lines (DSL).

[0118] In various examples, computer system 463 may include an image processor, image processing engine, media processor, or any specialized digital signal processor (DSP) used to process digital images. The image processor may use parallel computing using Single Instruction, Multiple Data (SIMD) or Multiple Instruction, Multiple Data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0119] The communications connection may refer to the hardware / software used to connect the network interface to the bus. The communications connection is shown internal to computer system 463 for clarity of illustration, but can also be external to computer system 463. For purposes of illustration only, the hardware / software required to connect to the network interface may include internal and external technologies such as modems, including regular telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0120] The surgical data network associated with the surgical hub system 460 may be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, allowing it to go from one device (or segment) to another and to cloud computing resources. An intelligent surgical data network includes additional features that allow traffic to pass through the monitored surgical data network and configure each port within the network hub 461 or network switch 462. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0121] Modular devices 1a-1n located in an operating room may be coupled to a modular communication hub 465. Network hub 461 and / or network switch 462 may be coupled to a network router 466 to connect devices 1a-1n to a cloud computing system 464 or a local computer system 463. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to local computer system 463 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be coupled to network switch 462. Network switch 462 may be coupled to network hub 461 and / or to network router 466 to connect devices 2a-2m to a cloud 464. Data associated with devices 2a-2m may be transferred to cloud computing system 464 via network router 466 for data processing and manipulation. Data associated with the devices 2a-2m may also be transferred to a local computer system 463 for local data processing and manipulation.

[0122] 4, a computing system such as surgical hub system 460 may include a modular communications hub 465 configured to connect modular devices (e.g., surgical devices) located within a medical facility to a cloud-based system (e.g., a cloud computing system 464 that may include a remote server 467 coupled to remote storage 468). The modular communications hub 465 and devices may be connected in a room within the medical facility specially equipped for surgical procedures. In one aspect, the modular communications hub 465 may include a network hub 461 and / or a network switch 462 in communication with a network router 466. The modular communications hub 465 may be coupled to a local computer system to provide local computer processing and data manipulation.

[0123] FIG. 5 illustrates a logic diagram of a control system 520 for a surgical instrument or tool according to one or more embodiments of the present disclosure. The surgical instrument or tool may be configurable. The surgical instrument may include surgical fasteners specific to the procedure at hand, such as an imaging device, a surgical stapler, an energy device, an endocutter device, etc. For example, the surgical instrument may include any of a power stapler, a power stapler generator, an energy device, an advanced energy device, an advanced energy jaw device, an endocutter clamp, an energy device generator, an operating room imaging system, a smoke evacuation device, a suction irrigation device, an insufflation system, etc. The system 520 may include control circuitry. The control circuitry may include a microcontroller 521 with a processor 522 and a memory 523. For example, one or more of sensors 525, 526, 527 provide real-time feedback to the processor 522. A motor 530 driven by a motor driver 529 operably couples a longitudinally movable displacement member to drive the I-beam knife element. The tracking system 528 may be configured to determine the position of the longitudinally movable displacement member. The position information may be provided to the processor 522, which may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the firing member, firing bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control the firing of the I-beam, the movement of the obturator tube, the rotation of the shaft, and the articulation. The display 524 may display various operating states of the instrument and may also include touch screen functionality for data entry. Information displayed on the display 524 may be overlaid with images acquired via the endoscopic imaging module.

[0124] Microcontroller 521 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In one aspect, main microcontroller 521 may be, for example, an LM4F230H5QR ARM Cortex-M4F Processor Core available from Texas Instruments, with on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2 KB of EEPROM, one or more PWM modules, one or more QEI analog, and / or one or more 12-bit ADCs with 12 analog input channels, details of which are available in the product datasheet.

[0125] The microcontroller 521 may include a safety controller, which includes two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller can be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0126] The microcontroller 521 may be programmed to perform various functions, such as precise control over the speed and position of the knife and articulation system. In one embodiment, the microcontroller 521 may include a processor 522 and memory 523. The electric motor 530 may be a brushed direct current (DC) motor with a gearbox and mechanical linkage to the articulation or knife system. In one embodiment, the motor driver 529 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be easily substituted for use in the tracking system 528 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication No. 2017 / 0296213, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," published October 19, 2017, which is incorporated herein by reference in its entirety.

[0127] The microcontroller 521 may be programmed to provide precise control over the velocity and position of the displacement members and articulation system. The microcontroller 521 may be configured to calculate a response in the microcontroller 521 software. The calculated response may be compared to the measured response of the actual system to obtain an "observed" response, which is used for actual feedback decision-making. The observed response may be a suitably adjusted value that balances the smooth, continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0128] The motor 530 may be controlled by a motor driver 529 and may be used by the surgical instrument or tool firing system. In various forms, the motor 530 may be a brushed DC drive motor having a maximum rotational speed of approximately 25,000 RPM. In some examples, the motor 530 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 529 may include, for example, an H-bridge driver including field-effect transistors (FETs). The motor 530 may be powered by a power supply assembly releasably attached to the handle assembly or tool housing to provide control power to the surgical instrument or tool. The power supply assembly may include a battery, which may include multiple battery cells connected in series, that may be used as a power source to power the surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be lithium-ion batteries, which may be connectable to and detachable from the power supply assembly.

[0129] The motor driver 529 may be the A3941, available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with external N-channel power metal-oxide semiconductor field-effect transistors (MOSFETs) specifically designed for inductive loads, such as brushed DC motors. The driver 529 may include an intrinsic charge pump regulator, which provides full (>10V) gate drive for battery voltages up to 7V, allowing the A3941 to operate with reduced gate drive down to 5.5V. A bootstrap capacitor may be used to provide the required battery supply voltage above the N-channel MOSFET. An internal charge pump for the high-side drive allows DC (100% duty cycle) operation. The full-bridge may be driven in fast or slow decay mode using diode or synchronous rectification. In slow decay mode, current recirculation is possible through either the high-side or low-side FET. The power FETs may be protected from shoot-through by a resistor-adjustable dead time. Integrated diagnostics indicate undervoltage, overtemperature, and power bridge faults and can be configured to protect the power MOSFETs under most short circuit conditions. Other motor drivers may be easily substituted for use in tracking system 528 with an absolute positioning system.

[0130] The tracking system 528 may include controlled motor drive circuitry including a position sensor 525 according to one aspect of the present disclosure. The position sensor 525 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member including a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a firing member that may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a firing bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may be used generally to refer to any movable member of a surgical instrument or tool, such as a drive member, firing member, firing bar, I-beam, or any element that can be displaced. In one aspect, a longitudinally movable drive member may be coupled to a firing member, firing bar, and I-beam. Thus, the absolute positioning system may actually track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member may be coupled to any suitable position sensor 525 for measuring linear displacement. Thus, the longitudinally movable drive member, firing member, firing bar, or I-beam, or combinations thereof, may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor.The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system with a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system with a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system with a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system with a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0131] The electric motor 530 may include a rotatable shaft operably interfaced with a gear assembly mounted for meshing engagement with a set of drive teeth, or rack, on the displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 525 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor arrangement may be connected to a linear actuator by a rack-and-pinion arrangement or to a rotary actuator by a spur gear or other connection. A power source may provide power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member with a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable firing member, a firing bar, an I-beam, or a combination thereof.

[0132] One revolution of the sensor element associated with the position sensor 525 may correspond to a linear longitudinal displacement d1 of the displacement member, where d1 is the linear longitudinal distance the displacement member travels from point "a" to point "b" after one revolution of the sensor element coupled to the displacement member. The sensor arrangement may be connected via a gear reduction such that the position sensor 525 completes one or more revolutions for a full stroke of the displacement member. The position sensor 525 may complete multiple revolutions for a full stroke of the displacement member.

[0133] A series of switches (where n is an integer greater than 1) may be used alone or in combination with a gear reduction to provide a unique position signal for two or more revolutions of the position sensor 525. The state of the switches may be fed back to the microcontroller 521, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 525 is provided to the microcontroller 521. The position sensor 525 of the sensor arrangement may comprise a magnetic sensor, an analog rotary sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0134] The position sensor 525 may comprise any number of magnetic sensing elements, such as magnetic sensors classified according to whether they measure the total magnetic field or vector components of the magnetic field. The technologies used to produce both types of magnetic sensors may encompass many aspects of physics and electronics. Technologies used to sense magnetic fields may include search coils, fluxgates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetodiodes, magnetotransistors, optical fiber, magneto-optical, and microelectromechanical systems-based magnetic sensors, among others.

[0135] The position sensor 525 of the tracking system 528 with an absolute positioning system may comprise a magnetic rotary absolute positioning system. The position sensor 525 may be implemented as an AS5055EQFT single-chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 525 interfaces with the microcontroller 521 to provide the absolute positioning system. The position sensor 525 may be a low-voltage, low-power component and may include four Hall-effect elements in an area of ​​the position sensor 525 that may be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on-chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement simple, efficient algorithms for calculating hyperbolic and trigonometric functions, requiring only addition, subtraction, bit shifting, and table lookup operations. The angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 521 through a standard serial communications interface, such as a serial peripheral interface (SPI) interface. The position sensor 525 may provide 12-bit or 14-bit resolution and may be an AS5055 chip provided in a small QFN 16-pin 4x4x0.85mm package.

[0136] The tracking system 528, which comprises an absolute positioning system, may include and / or be programmed to implement a feedback controller, such as a PID, state feedback, and adaptive controller. The power supply converts the signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 525, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors may include sensor arrangements such as those described in U.S. Pat. No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuitry to combine the calculated response with the measured response using algorithms such as weighted averages and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account properties such as mass, inertia, viscous friction, and induced drag in order to predict what the state and output of the physical system will be given knowledge of the input.

[0137] The absolute positioning system may provide the absolute position of the displacement member upon powering up of the instrument without forcing the displacement member to retract or advance to a reset (zero or home) position, as may be required with conventional rotary encoders that simply count the number of forward or backward steps taken by the motor 530 to infer the position of the device actuator, drive bar, knife, etc.

[0138] Sensor 526, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the amplitude of strain exerted on the anvil during clamping, which may be indicative of the closure force applied to the anvil. The measured strain may be converted to a digital signal and provided to processor 522. Alternatively, or in addition to sensor 526, sensor 527, such as a load sensor, may measure the closure force applied to the anvil by the closure drive system. For example, sensor 527, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sled, which is configured to cam the staple driver upward and drive the staples into deforming contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 531 may be used to measure the current drawn by motor 530. The force required to advance the firing member may correspond, for example, to the current drawn by motor 530. The measured force may be converted to a digital signal and provided to processor 522.

[0139] For example, a strain gauge sensor 526 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to grasped tissue by the end effector may include a strain gauge sensor 526, such as, for example, a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 526 can measure the amplitude or magnitude of strain exerted on the jaw members of the end effector during clamping, which can indicate tissue compression. The measured strain can be converted to a digital signal and provided to the processor 522 of the microcontroller 521. A load sensor 527 can measure the force used to operate the knife element, for example, to cut tissue captured between the anvil and staple cartridge. A magnetic field sensor can be used to measure the thickness of the captured tissue. The magnetic field sensor's measurements can also be converted to a digital signal and provided to the processor 522.

[0140] Measurements of tissue compression, tissue thickness, and / or force required to close the end effector on the tissue, measured by sensors 526, 527, respectively, may be used by microcontroller 521 to characterize a selected position of the firing member and / or a corresponding value of firing member velocity. In one example, memory 523 may store techniques, equations, and / or look-up tables that may be used by microcontroller 521 during evaluation.

[0141] The surgical instrument or tool control system 520 may also include wired or wireless communication circuitry for communicating with a surgical hub, such as surgical hub 460, as shown in FIG.

[0142] FIG. 6 illustrates an exemplary surgical system 680 according to the present disclosure and may include a surgical instrument 682 that can communicate with a console 694 or a portable device 696 through a local area network 692 and / or a cloud network 693 via wired and / or wireless connections. The console 694 and the portable device 696 may be any suitable computing devices. The surgical instrument 682 may include a handle 697, an adapter 685, and a loading unit 687. The adapter 685 releasably couples to the handle 697, and the loading unit 687 releasably couples to the adapter 685 such that the adapter 685 transmits force from the drive shaft to the loading unit 687. The adapter 685 or the loading unit 687 may include a force gauge (not explicitly shown) disposed therein to measure force applied to the loading unit 687. The loading unit 687 may include an end effector 689 having a first jaw 691 and a second jaw 690. The loading unit 687 may be an in vivo loading or multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 687 from the surgical site to reload it.

[0143] The first jaw 691 and the second jaw 690 may be configured to clamp tissue therebetween, fire fasteners through the clamped tissue, and sever the clamped tissue. The first jaw 691 may be configured to fire at least one fastener multiple times, or may be configured to contain a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that may be fired two or more times before being replaced. The second jaw 690 may include an anvil that deforms or otherwise secures fasteners as they are ejected from the multi-fire fastener cartridge.

[0144] The handle 697 may include a motor coupled to the drive shaft to affect rotation of the drive shaft. The handle 697 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, a touch screen, and any other suitable input mechanism or user interface that may be engaged by a clinician to activate the motor.

[0145] The control interface of the handle 697 may be in communication with a controller 698 of the handle 697 to selectively activate the motors to affect rotation of the drive shaft. The controller 698 may be disposed within the handle 697 and may be configured to receive input from the control interface and adapter data from the adapter 685 or loading unit data from the loading unit 687. The controller 698 may analyze the input from the control interface and data received from the adapter 685 and / or loading unit 687 to selectively activate the motors. The handle 697 may also include a display viewable by a clinician while using the handle 697. The display may be configured to display portions of the adapter or loading unit data before, during, or after firing of the instrument 682.

[0146] The adapter 685 may include an adapter identification device 684 disposed therein, and the loading unit 687 may include a loading unit identification device 688 disposed therein. The adapter identification device 684 may be in communication with a controller 698, and the loading unit identification device 688 may be in communication with the controller 698. It will be appreciated that the loading unit identification device 688 may be in communication with the adapter identification device 684, which relays or passes communications from the loading unit identification device 688 to the controller 698.

[0147] The adapter 685 may also include multiple sensors 686 (one shown) disposed about its periphery to detect various conditions of the adapter 685 or the environment (e.g., whether the adapter 685 is connected to the loading unit, whether the adapter 685 is connected to the handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain on the drive shaft, the temperature within the adapter 685, the number of times the adapter 685 has been fired, the peak force of the adapter 685 during firing, the total amount of force applied to the adapter 685, the peak retract force of the adapter 685, the number of times the adapter 685 has been paused during firing, etc.). The multiple sensors 686 may provide input to the adapter identification device 684 in the form of data signals. The data signals of the multiple sensors 686 may be stored in the adapter identification device 684 or may be used to update adapter data stored in the adapter identification device 684. The data signals of the multiple sensors 686 may be analog or digital. The multiple sensors 686 may include a force gauge that measures the force exerted on the loading unit 687 during firing.

[0148] The handle 697 and adapter 685 may be configured to interconnect the adapter identification device 684 and the loading unit identification device 688 with the controller 698 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface may be a contactless electrical interface for wirelessly transmitting (e.g., inductively transmitting) energy and signals therebetween. It is also contemplated that the adapter identification device 684 and the controller 698 may communicate wirelessly with each other via a wireless connection that is separate from the electrical interface.

[0149] The handle 697 may include a transceiver 683 configured to transmit instrument data from the controller 698 to other components of the system 680 (e.g., the LAN 20292, the cloud 693, the console 694, or the portable device 696). The controller 698 may also transmit instrument data and / or measurement data associated with one or more sensors 686 to the surgical hub. The transceiver 683 may receive data (e.g., cartridge data, loading unit data, adapter data, or other notifications) from the surgical hub 670. The transceiver 683 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 680. For example, the controller 698 may transmit instrument data to the console 694, including the serial number of an attached adapter (e.g., adapter 685) attached to the handle 697, the serial number of a loading unit (e.g., loading unit 687) attached to the adapter 685, and the serial number of a multi-fire fastener cartridge loaded in the loading unit. The console 694 may then return data associated with the attached cartridge, loading unit, and adapter (e.g., cartridge data, loading unit data, or adapter data), respectively, to the controller 698. The controller 698 may display a message on a local instrument display or may send a message via the transceiver 683 to the console 694 or portable device 696, respectively, to display the message on the display 695 or portable device screen.

[0150] 7A illustrates a surgical system 700 that may include a matrix of surgical information. This surgical information may include any discrete atoms of information related to a surgical procedure. Generally described, such surgical information may include information related to the context and scope of the surgical procedure itself (e.g., healthcare information 728). Such information may include data such as, for example, procedure data and patient record data. The procedure data and / or patient record data may be associated with an associated healthcare data system 716 that is in communication with the surgical hub 704.

[0151] The surgical information may include information related to the configuration and / or control of devices being used in the procedure (e.g., device operation information 729). Such device operation information 729 may include information regarding the initial configuration of a surgical device. Device operation information 729 may include information regarding changes to the configuration of a surgical device. Device operation information 729 may include information regarding controls sent from the surgical hub 704 to the device and the information flow associated with such controls.

[0152] Surgical information may include information generated during the surgery itself (e.g., surgical information 727). Such surgical information 727 may include any information generated by a surgical data source 726. The data source 726 may include any device within the surgical context that may generate useful surgical information 727. This surgical information 727 may present itself as an observable quality of the data source 726. The observable quality may include static qualities such as a device's model number, serial number, etc. The observable quality may include dynamic qualities such as the state of a device's configurable settings. Surgical information 727 may present itself, for example, as the result of sensor observations. Sensor observations may include observations from specific sensors in the operating room, sensors for monitoring conditions such as the patient's condition, sensors embedded in surgical devices, etc. Sensor observations may include information used during the surgery, such as video, audio, etc. Surgical information 727 may present itself as device event data. The surgical device may generate notifications and / or log events, and such events may be included in surgical information 727 for communication to the surgical hub 704. Surgical information 727 may present itself, for example, as a result of manual recording. A medical professional may record during a procedure by asking the patient to take notes, capturing still images from the display, etc.

[0153] The surgical data sources 726 may include, for example, modular devices (e.g., which may include sensors configured to detect parameters associated with the patient, the HCP, and the environment, and / or the modular devices themselves), local databases (e.g., a local EMR database containing patient records), patient monitoring devices (e.g., blood pressure (BP) monitors and electrocardiography (EKG) monitors), HCP monitoring devices, environmental monitoring devices, surgical instruments, surgical support equipment, etc.

[0154] The surgical hub 704 can be configured to derive contextual information about the surgical procedure from the data based on, for example, a particular combination of received data or a particular order in which data is received from the data sources 726. The contextual information inferred from the received data can include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure the surgeon is performing, the type of tissue being operated on, or the body cavity that is the target of the procedure. This ability, by some aspects of the surgical hub 704, to derive or infer information about the surgical procedure from the received data can be referred to as “situational awareness.” For example, the surgical hub 704 can incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 704 that derives contextual information about the surgical procedure from the received data and / or surgical planning information received from the edge computing system 714 or the healthcare data system 716 (e.g., an enterprise cloud server).

[0155] In operation, this matrix of surgical information may exist as one or more information flows. For example, surgical information may flow from a surgical data source 726 to the surgical hub 704. Surgical information may flow from the surgical hub 704 to a surgical data source 726 (e.g., a surgical device). Surgical information may flow between the surgical hub 704 and one or more healthcare data systems 716. Surgical information may flow between the surgical hub 704 and one or more edge computing devices 714.

[0156] The surgical information as presented in one or more information flows may be used in conjunction with one or more artificial intelligence (AI) systems to further enhance the operation of the surgical system 700. For example, a machine learning system, such as those described herein, may operate on one or more information flows to further enhance the operation of the surgical system 700.

[0157] 7B illustrates an exemplary computer-implemented surgical system 730 having multiple information flows 732. The surgical computing device 704 may communicate with and / or incorporate one or more surgical data sources. For example, the imaging module 733 (and endoscope) may exchange surgical information with the surgical computing device 704. Such information may include information from the imaging module 733 (and endoscope), such as video information, current settings, system status information, etc. The imaging module 733 may receive information from the surgical computing device 704, such as control information, configuration information, operational updates (software / firmware, etc.).

[0158] For example, the generator module 734 (and corresponding energy devices) may exchange surgical information with the surgical computing device 704. Such information may include information from the generator module 734 (and corresponding energy devices), such as electrical information (e.g., current, voltage, impedance, frequency, wattage), activity state information, sensor information such as temperature, current settings, system events, active duration, and startup timestamps. The generator module 734 may receive information from the surgical computing device 704, such as control information, configuration information, changes in the nature of visible and audible notifications to the medical professional (e.g., changes in the pitch, duration, and melody of an audible tone), electrical application profiles and / or application logic that may instruct the generator module to provide energy having a defined characteristic curve over the application time, operational updates (e.g., software / firmware), and the like.

[0159] For example, the smoke evacuator 735 may exchange surgical information with the surgical computing device 704. Such information may include information from the smoke evacuator 735 such as operational information (e.g., revolutions per minute), activity status information, sensor information such as temperature, current settings, system events, active duration, and boot timestamps. The smoke evacuator 735 may receive information from the surgical computing device 704 such as control information, configuration information, operational updates (software / firmware, etc.).

[0160] For example, the aspirate / irrigate module 736 may exchange surgical information with the surgical computing device 704. Such information may include information from the aspirate / irrigate module 736, such as operational information (e.g., liters per minute), activity status information, internal sensor information, current settings, system events, active duration, and startup timestamps. The aspirate / irrigate module 736 may receive information from the surgical computing device 704, such as control information, configuration information, operational updates (software / firmware, etc.).

[0161] For example, the communications module 739, the processor module 737, and / or the storage array 738 may exchange surgical information with the surgical computing device 704. In one example, the communications module 739, the processor module 737, and / or the storage array 738 may comprise all or part of the computing platform on which the surgical computing device 704 operates. In one example, the communications module 739, the processor module 737, and / or the storage array 738 may provide local computing resources to other devices in the surgical system 730. Information from the communications module 739, the processor module 737, and / or the storage array 738 to the surgical computing device 704 may include logical computing related reports such as processing load, processing power, process identification, CPU %, CPU time, threads, GPU %, GPU time, memory utilization, memory threads, memory ports, energy usage, bandwidth related information, packets in, packets out, data rates, channel utilization, buffer status, packet loss information, system events, and other status information. The communications module 739, processor module 737, and / or storage array 738 may receive information, such as control information, configuration information, operational updates (software / firmware, etc.), etc., from the surgical computing device 704. The communications module 739, processor module 737, and / or storage array 738 may also receive information from the surgical computing device 704 generated by another element or device of the surgical system 730. For example, data source information may be transmitted to and stored in the storage array. For example, the data source information may be processed by the processor module 737.

[0162] For example, the intelligent instrument 740 (with or without a corresponding display) may exchange surgical information with the surgical computing device 704. Such information may include information from the intelligent instrument 740 regarding the operation of the instrument, such as device electrical and / or mechanical information (e.g., current, voltage, impedance, frequency, wattage, torque, force, pressure, etc.), load status information (e.g., information regarding the identity, type, and / or status of reusables such as staple cartridges), clamping force, tissue compression pressure, and / or internal sensor information such as time, system events, active duration, and activation timestamp. The intelligent instrument 740 may receive information from the surgical computing device 704, such as control information, configuration information, changes in the nature of visible and audible notifications to the medical professional (e.g., changes in the pitch, duration, and melody of an audible tone), mechanical application profiles and / or application logic that may instruct the instrument's mechanical components to operate with defined characteristics (e.g., blade / anvil advancement speed, mechanical advantage, firing time, etc.), operational updates (software / firmware, etc.), and the like.

[0163] For example, the sensor module 741 may exchange surgical information with the surgical computing device 704. Such information may include information from the sensor module 741 about its sensor capabilities, such as the sensor results themselves, observation frequency and / or resolution, observation type, device alerts such as alerts for sensor failure, observations exceeding a defined range, observations exceeding an observable range, etc. The sensor module 741 may receive information from the surgical computing device 704, such as control information, configuration information, changes in the nature of the observations (e.g., frequency, resolution, observation type, etc.), triggers defining specific events for observations, on controls, off controls, data buffering, data pre-processing algorithms, operational updates (software / firmware, etc.), etc.

[0164] For example, the visualization system 742 may exchange surgical information with the surgical computing device 704. Such information may include information from the visualization system 742, such visualization data itself (e.g., still images, video, advanced spectral visualization, etc.), visualization metadata (e.g., visualization type, resolution, frame rate, encoding, bandwidth, etc.), etc. The visualization system 742 may receive information from the surgical computing device 704, such as control information, configuration information, changes in video settings (e.g., visualization type, resolution, frame rate, encoding, etc.), visual display overlay data, data buffering sizes, data pre-processing algorithms, operational updates (software / firmware, etc.), etc.

[0165] For example, the surgical robot 743 may exchange surgical information with the surgical computing device 704. Information from the surgical robot 743 may include any of the aforementioned information as it applies to robotic instruments, sensors, and devices. Information from the surgical robot 743 may also include information related to the robotic operation or control of such instruments, such as electrical / mechanical feedback of the robotic articulator, system events, system settings, mechanical resolution, control operation logs, articulator path information, etc. The surgical robot 743 may receive information from the surgical computing device 704, such as control information, configuration information, operation updates (software / firmware, etc.).

[0166] 7C illustrates an example information flow associated with multiple surgical computing systems 704a, 704b within a common environment. As the overall configuration of a computer-implemented surgical system (e.g., computer-implemented surgical system 750) changes (e.g., as data sources are added and / or removed from the surgical computing system), additional surgical information may be generated to reflect the changes. In this example, a second surgical computing system 704b (e.g., a surgical hub) may be added (along with a corresponding surgical robot) to surgical system 750 with existing surgical computing system 704a. The messaging flows described herein represent additional surgical information flow 755 (e.g., further integrated, analyzed, and / or processed according to algorithms such as machine learning algorithms) used as disclosed herein.

[0167] Here, two surgical computing systems 704a, 704b request permission from the surgeon for the second surgical computing system 704b (with a corresponding surgical robot 756) to take control of the operating room from the existing surgical computing system 704a. The second surgical computing system 704b presents control of the corresponding surgical robot 756, robotic visualization tower 758, Monohat tool 759, and robotic stapler 749 in the operating room. Permission may be requested through the surgeon interface or console 751. Once permission is granted, the second surgical computing system 704b sends a message to the existing surgical computing system 704a requesting transfer of control of the operating room.

[0168] In one example, the surgical computing systems 704a, 704b can negotiate the nature of their interaction without external input based on previously collected data. For example, the surgical computing systems 704a, 704b may collectively determine that an upcoming surgical task requires the use of a robotic system. Such a determination may cause the existing surgical computing system 704a to autonomously hand over control of the operating room to a second surgical computing system 704b. Upon completion of the surgical task, the second surgical computing system 704b may then autonomously return control of the operating room to the existing surgical computing system 704a.

[0169] As illustrated in Figure 7C, the existing surgical computing system 704a has transferred control to a second surgical computing system 704b, which also assumes control of the surgeon interface 751 and secondary display 752. The second surgical computing system 704b assigns new identification numbers to the newly transferred devices. The existing surgical computing system 704a retains control of the handheld stapler 753, handheld powered dissector 754, and visualization tower 757. In addition, the existing surgical computing system 704a may perform a support role, with the processing and storage capabilities of the existing surgical computing system 704a now available to the second surgical computing system 704b.

[0170] 7D illustrates an exemplary surgical information flow in the context of a surgical procedure and corresponding exemplary uses of the surgical information for predictive modeling. The surgical information disclosed herein may provide data regarding one or more surgical procedures, including surgical tasks, instruments, instrument settings, motion information, procedural variations, and corresponding desirable metrics such as improved patient outcomes, lower costs (e.g., fewer resources utilized, shorter surgical time, etc.). The surgical information disclosed herein (e.g., that disclosed with respect to FIGS. 7A-7C ), in the context of one or more surgical systems and devices disclosed herein, provides a platform upon which certain machine learning algorithms and techniques disclosed herein may be used.

[0171] Surgical information 762 from multiple surgical procedures 764 (e.g., a subset of surgical information from each procedure) may be collected. Surgical information 762 may be collected from multiple surgical procedures 764, for example, by collecting data represented by one or more information flows disclosed herein.

[0172] To illustrate, an exemplary instance of surgical information 766 may be generated from an exemplary procedure 768 (e.g., a lung segmentectomy procedure as shown on timeline 769). Surgical information 766 may be generated during preoperative planning and may include patient record information. Surgical information 766 may be generated from data sources (e.g., data sources 726) during the course of a surgical procedure, including data generated each time medical personnel utilize a modular device paired with the surgical computing system (e.g., surgical computing system 704). The surgical computing system may receive this data from the paired modular device and other data sources. The surgical computing system itself may generate surgical information as part of its operation during a procedure. For example, the surgical computing system may record information related to configuration and control operations. The surgical computing system may record information related to situational awareness activities. For example, the surgical computing system may record recommendations, prompts, and / or other information provided to the medical team (e.g., provided via a display screen) that may be related to the next procedure step. For example, the surgical computing system may record configuration and control changes (e.g., adjustments to modular devices based on context) that may include activating a monitor, adjusting the field of view (FOV) of a medical imaging device, changing the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument, etc.

[0173] Hospital personnel retrieve the patient's EMR from the hospital's EMR database at 770. Based on the selected patient data in the EMR, the surgical computing system determines that the procedure to be performed is a thoracic procedure.

[0174] At 771, personnel scan incoming medical supplies for a procedure. The surgical computing system may cross-reference the scanned supplies with a list of supplies utilized in various types of procedures. The surgical computing system may verify that the mix of supplies corresponds to a thoracic procedure. Additionally, the surgical computing system may determine that the procedure is not a wedge resection (because the incoming supplies either do not include certain supplies needed for a thoracic wedge resection or are otherwise not compatible with a thoracic wedge resection). The medical personnel may scan a patient band via a scanner communicatively connected to the surgical computing system. The surgical computing system may verify the patient's identity based on the scanned data.

[0175] At 774, medical personnel turn on auxiliary equipment. The auxiliary equipment utilized may vary according to the type of surgical procedure and the technique used by the surgeon. In this example, the auxiliary equipment may include a smoke evacuator, an aspirator, and a medical imaging device. Once activated, the auxiliary equipment may pair with the surgical computing system. The surgical computing system may derive contextual information regarding the surgical procedure based on the paired type. In this example, the surgical computing system determines that the surgical procedure is a VATS procedure based on this particular combination of paired devices. The contextual information regarding the surgical procedure may be ascertained by the surgical computing system via information from the patient's EMR.

[0176] The surgical computing system may retrieve the steps of the procedure to be performed. For example, the steps may be associated with a treatment plan (e.g., a treatment plan specific to this patient's surgery, a treatment plan associated with a particular surgeon, a treatment plan template for the procedure in general, etc.).

[0177] At 775, staff attach EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices pair with the surgical computing system. The surgical computing system may receive data from the patient monitoring devices.

[0178] At 776, medical personnel induce anesthesia in the patient. The surgical computing system may record information related to this procedure step, such as data from the modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof.

[0179] At 777, the lung of the patient undergoing surgery is collapsed (and ventilation may be switched to the contralateral lung). The surgical computing system may determine that this procedure step has begun and may collect surgical information accordingly, including, for example, ventilator data, one or more timestamps, etc.

[0180] At 778, a medical imaging device (e.g., a scope) is inserted and video from the medical imaging device is initiated. The surgical computing system may receive medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. The data from the medical imaging device may include imaging data and / or imaging metadata, such as the angle at which the medical imaging device is oriented relative to visualization of the patient's anatomy, the number of medical imaging devices currently active, etc. The surgical computing system may record positioning information of the medical imaging device. For example, one technique for performing a VATS lobectomy places the camera in the anterior-inferior corner of the patient's thoracic cavity above the diaphragm. Another technique for performing a VATS segmentectomy places the camera in an anterior intercostal position relative to the segmental fissure.

[0181] For example, using pattern recognition or machine learning techniques, the surgical computing system may be trained to recognize the positioning of the medical imaging device according to the visualization of the patient's anatomy. For example, one technique for performing a VATS lobectomy utilizes a single medical imaging device. Another technique for performing a VATS segmentectomy uses multiple cameras. Yet another technique for performing a VATS segmentectomy uses an infrared light source (which may be communicatively coupled to the surgical computing system as part of the visualization system).

[0182] At 779, the surgical team begins the incision step of the procedure. The surgical computing system may collect data from the RF or ultrasonic generator indicating that the energy instrument is being fired. The surgical computing system may cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being fired at this point in the process (i.e., after previously discussed steps of the procedure have been completed) corresponds to the incision step. In one example, the energy instrument may be an energy tool mounted on a robotic arm of a robotic surgical system.

[0183] At 780, the surgical team proceeds to the ligation step of the procedure. The surgical computing system may collect surgical information 766 related to the surgeon ligating the arteries and veins based on receiving data from the surgical stapling and severing instrument indicating that such instrument is being fired. Next, the segmentectomy portion of the procedure is performed. The surgical computing system may collect information related to the surgeon transecting the parenchyma. For example, the surgical computing system may receive surgical information 766 from the surgical stapling and severing instrument, including data related to its cartridge, settings, firing details, etc.

[0184] At 782, the node dissection step is then performed. The surgical computing system may collect surgical information 766 regarding the surgical team dissecting the node and performing the leak test. For example, the surgical computing system may collect data received from the generator indicating that an RF or ultrasonic instrument is being fired, including electrical and status information associated with the firing. The surgeon periodically alternates between the surgical stapling / cutting instrument and the surgical energy (i.e., RF or ultrasonic) instrument depending on the particular step in the procedure. The surgical computing system may collect surgical information 766 taking into account the particular sequence in which the stapling / cutting instrument and the surgical energy instrument are used. In one example, a robotic tool may be used for one or more steps in the surgical procedure. The surgeon may, for example, alternate between using the robotic tool and a handheld surgical instrument and / or use the devices simultaneously.

[0185] The incisions are then closed and the post-operative portion of the procedure begins. The patient is de-anesthetized at 784. The surgical computing system may collect surgical information regarding the patient emerging from anesthesia, for example, based on ventilator data (i.e., the patient's breathing rate begins to increase).

[0186] At 785, medical personnel remove various patient monitoring devices from the patient. The surgical computing system may collect information regarding the outcome of the procedure. For example, the surgical computing system may collect information related to the loss of EKG, BP, and other data from the patient monitoring devices.

[0187] The surgical information 762 (including the surgical information 766) may be structured and / or labeled. The surgical computing system may inherently provide such structure and / or labeling in the data collection. For example, the surgical information 762 may be labeled according to particular characteristics, desired results (e.g., efficiency, patient outcome, cost, and / or combinations thereof, etc.), particular surgical techniques, aspects of instrumentation (e.g., surgical instrument selection, timing, and activation, instrument settings, nature of instrument use, etc.), identities of medical professionals involved, particular patient characteristics, etc., each of which may be present in the data collection.

[0188] The surgical information (e.g., surgical information 762 collected over the procedure 764) may be used in connection with one or more artificial intelligence (AI) systems. AI may be used to perform computer cognitive tasks. For example, AI may be used to perform complex tasks based on observation of data. AI may be used to enable computing systems to perform cognitive tasks and solve complex tasks. AI may include using machine learning and machine learning techniques. ML techniques may include, for example, performing complex tasks without being programmed (e.g., explicitly programmed). For example, ML techniques may improve over time based on completing tasks with different inputs. An ML process may train itself, for example, using input data and / or a training dataset.

[0189] Machine learning (ML) techniques may be used, for example, in the medical field. For example, ML may be used on a set of data (e.g., a set of surgical data) to generate output (e.g., reduced surgical data, processed surgical data). In an example, the output of the ML process may include identified trends or relationships in the data input for processing. The output may include verifying results and / or outcomes associated with the input data. In an example, the input to the ML process may include medical data such as surgical images and patient scans. The ML process may output a determined medical condition based on the input surgical images and patient scans. The ML process may be used to diagnose a medical condition, for example, based on the surgical scans.

[0190] An ML process may improve itself, for example, using historical data and / or input data that trained the ML process. Thus, an ML process may continually improve with added inputs and processing. The ML process may update based on the input data. For example, over time, an ML process that generates medical outcomes based on medical data may improve and become more accurate and consistent in medical diagnoses.

[0191] ML processes may be used to solve different complex tasks (e.g., medical tasks). For example, ML processes may be used for data reduction, data preparation, data processing, trend identification, outcome determination, medical diagnosis, and / or the like. For example, an ML process may take surgical data as input and process the data for use in medical analysis. The processed data may be used to determine a medical diagnosis. Finally, an ML process may take raw surgical data and generate useful medical information (e.g., medical trends and / or diagnoses) associated with the raw surgical data.

[0192] ML processes may be combined to perform different discrete tasks on an input data set. For example, an ML process may include testing different combinations of ML subprocesses performing discrete tasks to determine which combination performs best (e.g., competitive use of different process / algorithm types and training to determine the best combination for a data set). For example, an ML process may include subprocess (e.g., algorithm) control and monitoring to refine and / or validate results and / or outcomes (e.g., error bounds).

[0193] An ML process may be initialized and / or set up to perform a task. For example, the ML process may be initialized based on initialization configuration information. The initialized ML process may be an untrained ML process and / or a base ML process for performing the task. An untrained ML process may be inaccurate in performing a specified task. As the ML process is trained, the task may be performed more accurately.

[0194] The initialization configuration information for the ML process may include initial settings and / or parameters. For example, the initial settings and / or parameters may include defined ranges for use by the ML process. The ranges may include manually entered and / or received data ranges. The ranges may include default ranges and / or randomized ranges for unreceived variables that may be used, for example, to complete the dataset for processing. For example, if a dataset lacks a data range, the default data range may be used as a substitute to run the ML process.

[0195] The initialization configuration information for the ML process may include data storage locations. For example, locations or data storage and / or databases associated with data interactions may be included. The databases associated with data interactions may be used to identify trends in the dataset. The databases associated with data interactions may include mappings of data to medical conditions. For example, the databases associated with data interactions may include mappings of heart rate data to arrhythmias, etc.

[0196] The initialization configuration information may include parameters associated with defining the system. The initialization configuration information may include instructions (e.g., methods) associated with displaying, verifying, and / or providing information to a user. For example, the initialization configuration may include instructions for an ML process to output data in a particular format for visualization to a user.

[0197] ML techniques may be used, for example, to perform data reduction. ML techniques for data reduction may include using multiple different data reduction techniques. For example, ML techniques for data reduction may include using one or more of the following: CUR matrix decomposition; decision trees; expectation-maximization (EM) processes (e.g., algorithms); explicit semantic analysis (ESA); exponential smoothing forecasting; generalized linear models; k-means clustering (e.g., nearest neighbor); naive Bayes; neural network processes; multivariate analysis; o-cluster; singular value decomposition; Q-learning; temporal difference (TD); deep adversarial networks; support vector machines (SVM); linear regression; dimensionality reduction; linear discriminant analysis (LDA); adaptive boosting (e.g., AdaBoost); gradient descent (e.g., stochastic gradient descent (SGD)); outlier detection; and / or others.

[0198] ML techniques may be used to perform data reduction, for example, using CUR matrix decomposition. CUR matrix decomposition may include using a matrix decomposition model (e.g., process, algorithm), such as a low-rank matrix decomposition model. For example, CUR matrix decomposition may include a low-rank matrix decomposition process that is expressed (e.g., explicitly expressed) in several (e.g., a small number) columns and / or rows of a data matrix (e.g., the CUR matrix decomposition may be interpretable). CUR matrix decomposition may include selecting columns and / or rows associated with statistical leverage and / or large influence in the data matrix. Using CUR matrix decomposition may enable identifying attributes and / or rows within the data matrix. Simplification of larger datasets (e.g., using CUR matrix decomposition) may enable users to review and interact (e.g., with the data). CUR matrix decomposition may facilitate regression, classification, clustering, and / or other processes.

[0199] ML techniques may be used to perform data reduction using, for example, decision trees (e.g., decision tree models). Decision trees may be used, for example, as a framework for quantifying outcome values ​​and / or the probability of an outcome occurring. Decision trees may be used, for example, to calculate values ​​for uncertain outcome nodes (e.g., in a decision tree). Decision trees may be used, for example, to calculate values ​​for decision nodes (e.g., in a decision tree). Decision trees may be models that enable classification and / or regression (e.g., applicable to classification and / or regression problems). Decision trees may be used to analyze numerical (e.g., continuous) and / or categorical data. Decision trees may be more successful and / or more efficient with large datasets (e.g., compared to other data reduction techniques).

[0200] Decision trees may be used in combination with other decision trees. For example, a random forest may refer to a collection of decision trees (e.g., an ensemble of decision trees). A random forest may include a collection of decision trees whose results may be aggregated into a result. A random forest may be a supervised learning algorithm. A random forest may be trained, for example, using a bagging training process.

[0201] A random decision forest (e.g., random forest) may add randomness (e.g., additional randomness) to a model, for example, while growing a tree. A random forest may be used, for example, to search for the best feature among a random subset of features, rather than searching for the most important feature (e.g., while splitting a node). Searching for the best feature among a random subset of features may result in a wide variety, which may result in a better (e.g., more efficient and / or accurate) model.

[0202] Random forests may include using parallel ensembles. Parallel ensembles may include, for example, fitting (e.g., several) decision tree classifiers in parallel on different dataset subsamples. Parallel ensembles may include using majority voting or averaging on the results or final outcome. Parallel ensembles may be used to minimize overfitting and / or increase prediction accuracy and control. Random forests with multiple decision trees may (e.g., generally) be more accurate than single decision tree-based models. A set of decision trees with controlled variation may be constructed, for example, by combining bootstrap aggregation (e.g., bagging) with random feature selection.

[0203] ML techniques may be used to perform data reduction, for example, using an expectation-maximization (EM) model (e.g., process, algorithm). For example, an EM model may be used to find likelihood (e.g., local maximum likelihood) parameters of a statistical model. An EM model may be used when equations cannot be solved directly. An EM model may consider latent variables and / or unknown parameters and known data observations. For example, an EM model may determine that missing values ​​are present in a dataset. An EM model receives configuration information indicating to assume the presence of missing (e.g., unobserved) data points in the dataset.

[0204] The EM model may use component clustering. For example, component clustering may allow EM components to be grouped into high-level clusters. For example, if component clustering is disabled (e.g., in the EM model), the components may be treated as clustered.

[0205] ML techniques may be used to perform data reduction, for example, using Explicit Semantic Analysis (ESA). ESA may be used at the level of semantics (e.g., meaning) rather than the vocabulary (e.g., surface form vocabulary) of words or documents. ESA may focus on the meaning of a set of text, for example, as combinations of concepts found within the text. ESA may be used for document classification. ESA may be used for semantic relevance computation (e.g., how similar words or fragments of text are to each other). ESA may be used for information retrieval.

[0206] ESAs may be used, for example, for document classification. Document classification may include tagging documents for management and sorting. Tagging documents (e.g., with keywords) may make them easier to search. Keyword tagging (e.g., using only keyword tagging) may limit the accuracy and / or efficiency of document classification. For example, using keyword tagging may reveal (e.g., only) documents that have the keyword, but not documents that have words with similar meanings to the keyword. Semantically classifying text (e.g., using ESAs) may improve a model's understanding of the text. Semantically classifying text may include representing documents as concepts and reducing reliance on specific keywords.

[0207] ML techniques may be used to perform data reduction, for example, using an exponential smoothing forecasting model. Exponential smoothing may be used to smooth time series data, for example, using an exponential window function. For example, in a moving average, past observations may be weighted equally, but using an exponential function, weights may be assigned that decrease exponentially over time.

[0208] ML techniques may be used to perform data reduction, for example, using linear regression. Linear regression may be used to predict continuous outcomes. For example, linear regression may be used to predict the value of a variable (e.g., a dependent variable) based on the values ​​of different variables (e.g., independent variables). Linear regression may apply a linear approach to model the relationship between a scalar response and one or more explanatory variables (e.g., a dependent variable and / or independent variables). Simple linear regression may refer to a linear regression use case associated with one explanatory variable. Multiple linear regression may refer to a linear regression use case associated with two or more explanatory variables. Linear regression may model the relationship, for example, using a linear prediction function. The linear prediction function may estimate unknown model parameters from a dataset.

[0209] For example, linear regression may be used to identify patterns within a training dataset. The identified patterns may relate to groupings of values ​​and / or labels. The model may learn the relationship between (e.g., each) label and expected outcomes. After training, the model may be used on raw data outside the training dataset (e.g., data that does not have a mapped and / or known output). A trained model using linear regression may determine a calculated prediction associated with the raw data, such as identifying seasonal changes in sales data.

[0210] ML techniques may be used to perform data reduction, for example, generalized linear models (GLMs). GLMs may be used as flexible generalizations of linear regression. GLMs may generalize linear regression, for example, by allowing linear models to relate to response variables.

[0211] ML techniques may be used to perform data reduction, for example, using k-means clustering (e.g., nearest neighbor models). K-means clustering may be used in vector quantization. K-means clustering may be used in signal processing. K-means clustering may, for example, aim to divide n observations into k clusters, with each observation falling into the cluster with the closest mean.

[0212] K-means clustering may include K-Nearest Neighbor (KNN) learning. KNN may be instance-based learning (e.g., non-generalized learning, lazy learning). KNN may refrain from building a general internal model. KNN may include storing instances corresponding to training data in an n-dimensional space. KNN may use the data to classify data points, for example, based on a similarity measure (e.g., a Euclidean distance function). Classification may be calculated, for example, based on a majority vote of the k neighbors of a point (e.g., each point). KNN may be robust to noisy training data. Accuracy may depend on data quality (e.g., for KNN). KNN may include selecting the number of neighbors to consider (e.g., an optimal number of neighbors to consider). KNN may be used for classification and / or regression.

[0213] ML techniques may be used to perform data reduction, for example, using a naive Bayes model (e.g., process). For example, a naive Bayes model may be used to build a classifier. Using the naive Bayes model, a class label may be assigned to a problem instance (e.g., represented as a vector of feature values). The class label may be drawn from a set (e.g., a finite set). Different processes (e.g., algorithms) may be used to train the classifier. A family of processes (e.g., a family of algorithms) may be used. A family of processes may be based on the principle that a naive Bayes classifier (e.g., all naive Bayes) classifier assumes that the value of a feature is independent of the value of different features (e.g., given a class variable).

[0214] ML techniques may be used to perform data reduction, for example, using neural networks. The neural network may learn (e.g., be trained) by processing examples, for example, to perform other tasks (e.g., similar tasks). The processed examples may include inputs and results (e.g., inputs mapped to results). The neural network may learn by forming probability-weighted associations between inputs and results. The probability-weighted associations may be stored in the neural network's data structure. Training the neural network from a given example may be performed by determining the difference between the network's processed output (e.g., prediction) and a target output. This difference may be an error. The neural network may adjust the weighted associations (e.g., stored weighted associations), for example, according to a learning rule and an error value.

[0215] ML techniques may be used to perform data reduction, for example using multivariate analysis, which may include performing multivariate state estimation and / or non-negative matrix factorization.

[0216] ML techniques may be used to perform data reduction using, for example, a support vector machine (SVM). SVM may be used in multidimensional spaces (e.g., high-dimensional spaces, infinite-dimensional spaces). SVM may be used to construct a hyperplane (e.g., a set of hyperplanes). A hyperplane with the greatest distance (e.g., compared to other constructed hyperplanes) from the nearest training data point within a class (e.g., any class) may achieve strong separation (e.g., generally, the larger the margin, the lower the generalization error of the classifier). SVM may be effective in high-dimensional spaces. SVM may behave differently, for example, based on different mathematical functions (e.g., kernels, kernel functions). For example, the kernel function may include one or more of linear, polynomial, radial basis function (RBF), sigmoid, etc. Kernel functions may be used as SVM classifiers. SVM may be limited, for example, in use cases where the dataset contains a large amount of noise (e.g., overlapping target classes).

[0217] ML techniques may be used to perform data reduction, such as, for example, dimensionality reduction. Reducing the dimensionality of a sample of data (e.g., unlabeled data) may help refine groups and / or clusters. Reducing the number of variables in a model may simplify the data's trends. Simplified data trends may allow for more efficient processing. Dimensionality reduction may be used, for example, when many (e.g., too many) dimensions obscure (e.g., adversely affect) insights, trends, patterns, outcomes, and / or the like.

[0218] Reducing dimensionality may include using principal component analysis (PCA). PCA may be used to establish principal components that govern the relationships between data points. PCA may focus on simplifying (e.g., simplifying only) the principal components. Dimensionality reduction (e.g., PCA) may be used to maintain the diversity of data groupings within a dataset, but rationalize the number of distinct groups.

[0219] ML techniques may be used to perform data reduction, e.g., linear discriminant analysis (LDA). LDA may refer to a linear decision boundary classifier, which may be created, for example, by fitting class conditional densities to data (e.g., and applying Bayes' rule). LDA may include a generalization of Fisher's Linear Discriminant (e.g., projecting a given dataset into a lower-dimensional space to reduce dimensionality, minimize model complexity, and reduce computational cost). An LDA model (e.g., a standard LDA model) may fit classes with Gaussian densities. An LDA model may assume that classes (e.g., all classes) share a covariance matrix. LDA may be similar to an analysis of variance (ANOVA) process and / or regression analysis. For example, LDA may be used to express a dependent variable as a linear combination of other features and / or measurements.

[0220] ML techniques may be used to perform data reduction, such as, for example, adaptive boosting (e.g., AdaBoost). Adaptive boosting may include creating a classifier (e.g., a powerful classifier). Adaptive boosting may include creating a classifier by combining multiple classifiers (e.g., poorly performing classifiers), for example, to obtain a resulting classifier with high accuracy. AdaBoost may be an adaptive classifier that improves the efficiency of a classifier. AdaBoost may trigger overfitting. AdaBoost may be used (e.g., most commonly used) to improve the performance of decision trees, base estimators, binary classification problems, and / or the like. AdaBoost may be sensitive to noisy data and / or outliers.

[0221] ML techniques may be used to perform data reduction, such as stochastic gradient descent (SGD). SGD may include an iterative process used to optimize a function (e.g., an objective function). SGD may be used, for example, to optimize an objective function with specific smoothness properties. Stochastic may refer to random probability. SGD may be used, for example, to reduce computational load in high-dimensional optimization problems. SGD may be used, for example, to enable faster iterations while trading off a slower convergence rate. Gradient may refer, for example, to the slope of a function that calculates the degree of change of a variable in response to a change in another variable. Gradient descent may refer to a convex function that outputs the partial derivative of a set of its input parameters. For example, α may be a learning rate, and J may be the cost of training examples for the i-th iteration. This formula may represent a stochastic gradient descent weight update method for the j-th iteration. In large-scale ML and sparse ML, SGD may be applied to problems in text classification and / or natural language processing (NLP). SGD can be sensitive to feature scaling (e.g., it may be necessary to use a range of hyperparameters, such as the regularization parameter and number of iterations).

[0222] ML techniques may be used to perform data reduction, such as using outlier detection. An outlier may be a data point that contains information (e.g., useful information) about the abnormal behavior of the system described by the data. Outlier detection processes may include univariate and multivariate processes.

[0223] The ML process may be trained, for example, using one or more training methods. For example, the ML process may be trained using one or more of the following training techniques: supervised learning; unsupervised learning; semi-supervised learning; reinforcement learning; and / or others.

[0224] Machine learning can be supervised (e.g., supervised learning). A supervised learning algorithm can create a mathematical model from training a dataset (e.g., training data). FIG. 8A illustrates an exemplary supervised learning framework 800. The training data (e.g., training examples 802 as shown in FIG. 8) may consist of a set of training examples (e.g., input data mapped to labeled outputs as shown in FIG. 8). The training examples 802 may include one or more inputs and one or more labeled outputs. The labeled outputs may serve as supervisory feedback. In the mathematical model, the training examples 802 may be represented by an array or vector, sometimes called a feature vector. The training data may be represented by rows of the feature vector that form a matrix. Through iterative optimization of an objective function (e.g., a cost function), a supervised learning algorithm can learn a function (e.g., a prediction function) that can be used to predict an output associated with one or more new inputs. A properly trained predictive function (e.g., a trained ML model 808) may determine outputs 804 (e.g., labeled outputs) for one or more inputs 806 that may not be part of the training data (e.g., input data that does not have a mapped, labeled output, as shown in FIG. 8). Exemplary algorithms may include linear regression, logistic regression, neural networks, nearest neighbors, naive Bayes, decision trees, SVMs, and / or others. Exemplary problems that can be solved by supervised learning algorithms may include classification, regression problems, etc.

[0225] Machine learning can be unsupervised (e.g., unsupervised learning). FIG. 8B illustrates an exemplary unsupervised learning framework 810. An unsupervised learning algorithm 814 may train on a dataset that may include input 811 and may find structure 812 in the data (e.g., pattern detection and / or descriptive modeling). The structure 812 in the data may resemble groupings or clusterings of data points. Thus, the algorithm 814 may learn from training data that may be unlabeled. Instead of responding to supervised feedback, the unsupervised learning algorithm may identify commonalities in the training data and may react based on the presence or absence of such commonalities in each training data. For example, training may include operating on training input data to generate a model and / or output with a particular energy (e.g., cost function, etc.), and such energy may be used to further refine the model (e.g., to define a model that minimizes the cost function given the training input data). Exemplary algorithms may include the Apriori algorithm, K-means, K-nearest neighbors (KNN), K-medians, etc. Representative problems that can be solved by unsupervised learning algorithms may include clustering problems, anomaly / outlier detection problems, etc.

[0226] Machine learning may be semi-supervised (e.g., semi-supervised learning). Semi-supervised learning algorithms may be used in scenarios where labeling data is costly (e.g., because a skilled expert is required to label the data) and labels for the data are limited. Semi-supervised learning models may take advantage of the idea that while the group membership of unlabeled data is unknown, the data still holds important information about group parameters.

[0227] Machine learning may include reinforcement learning, which may be an area of ​​machine learning that may concern how a software agent can take actions in an environment to maximize some notion of cumulative reward. Reinforcement learning algorithms may not assume knowledge of an exact mathematical model of the environment (e.g., represented by a Markov decision process (MDP)) and may be used when an exact model may not be feasible. Reinforcement learning algorithms may be used in autonomous vehicles or in learning to play games against human opponents. Exemplary algorithms may include Q-learning, temporal difference (TD), deep adversarial networks, and / or others.

[0228] Reinforcement learning may involve an algorithm (e.g., an agent) that continuously learns from an environment in an iterative manner. During the training process, the agent may learn from experience with the environment until the agent has explored the full range of states (e.g., possible states). Reinforcement learning may be defined by the type of problem. Reinforcement learning solutions may be classified as reinforcement learning algorithms. In a problem, an agent may determine an action to select (e.g., a best action) based on the agent's current state. When steps are repeated, the problem may be referred to as an MDP.

[0229] For example, reinforcement learning may include an action step. The action step in reinforcement learning may include an agent observing an input state. The action step in reinforcement learning may include causing an agent to perform an action using a decision-making function. The action step may include an agent receiving a reward and / or reinforcement from the environment (e.g., after an action is performed). The action step in reinforcement learning may include storing state-action pair information related to the reward.

[0230] Machine learning may be part of a technology platform called cognitive computing (CC), which may comprise various fields such as computer science and cognitive science. CC systems may be able to learn at scale, reason purposefully, and interact naturally with humans. Self-teaching algorithms, which may use data mining, visual recognition, and / or natural language processing, may enable CC systems to solve problems and optimize human processes.

[0231] The output of a machine learning training process may be a model for predicting outcomes for new data sets. For example, a linear regression learning algorithm may have a cost function that can minimize the prediction error of a linear prediction function during the training process by adjusting the coefficients and constants of the linear prediction function. If a minimum value can be reached, the linear prediction function with the adjusted coefficients may be considered trained and constitute the model produced by the training process. For example, a neural network (NN) algorithm for classification (e.g., a multilayer perceptron (MLP)) may include a hypothesis function represented by a network of layers of nodes with assigned biases and interconnected by weighted connections. The hypothesis function may also be a nonlinear function (e.g., a highly nonlinear function) that may include linear and logistic functions nested together, with an outermost layer consisting of one or more logistic functions. The NN algorithm may include a cost function for minimizing classification error by adjusting biases and weights through a process of feedforward and backpropagation. If a global minimum can be reached, the optimized hypothesis function with its adjusted bias and weight layers may be considered trained and constitute the model that the training process produced.

[0232] Data aggregation may be performed for machine learning as a first stage in a machine learning lifecycle. Data aggregation may include steps such as identifying various data sources, collecting data from the data sources, and integrating the data. For example, to train a machine learning model for predicting surgical complications and / or post-surgical recovery rates, data sources including pre-surgical data such as a patient's medical condition and biomarker measurement data may be identified. Such data sources may be a patient's electronic medical record (EMR), a computing system that stores the patient's pre-surgical biomarker measurement data, and / or other similar data stores. Data from such data sources may be retrieved and stored in a central location for further processing in the machine learning lifecycle. Data from such data sources may be linked (e.g., logically linked) and accessed as if they were centrally stored. Surgical procedure data and / or post-surgical procedure data may be similarly identified and collected. Furthermore, the collected data may be integrated. In examples, a patient's pre-surgical medical record data, pre-surgical biomarker measurement data, pre-surgical data, surgical data, and / or post-surgical data may be combined into a patient record, which may be an EMR.

[0233] Data preparation can be performed for machine learning as another stage of the machine learning lifecycle. Data preparation can include data preprocessing steps such as data formatting, data cleaning, and data sampling. For example, collected data may not be in a data format suitable for training a model. Such data records can be converted into a flat file format for model training. Such data can be mapped to numerical values ​​for model training. Such identifying data can be removed before model training. For example, identifying data can be removed for privacy reasons. As another example, data can be removed because there may be more available data than can be used for model training. In such cases, a subset of the available data can be randomly sampled and selected for model training, and the remainder can be discarded.

[0234] Data preparation may include data transformation operations (e.g., after preprocessing), such as scaling and aggregation. For example, the preprocessed data may include data values ​​at various scales. These values ​​may be scaled up or down, e.g., to be between 0 and 1, for model training. For example, the preprocessed data may include data values ​​that have more meaning when aggregated.

[0235] Model training may be another aspect of the machine learning life cycle. The model training process described herein may depend on the machine learning algorithm used. A model may be considered suitably trained after it has been trained, cross-validated, and tested. Thus, a dataset from the data preparation stage (e.g., an input dataset) may be divided into a training dataset (e.g., 60% of the input dataset), a validation dataset (e.g., 20% of the input dataset), and a test dataset (e.g., 20% of the input dataset). After a model is trained on the training dataset, it may be run on the validation dataset to reduce overfitting. If the model's accuracy is increasing, but decreases when run on the validation dataset, this may indicate an overfitting problem. The test dataset may be used to test the accuracy of the final model to determine whether it is ready for deployment or whether more training may be required.

[0236] Model deployment can be another aspect of the machine learning lifecycle. Models may be deployed as part of a standalone computer program. Models may be deployed as part of a larger computing system. Models may be deployed with model performance parameters. Such performance parameters may monitor model accuracy as it is used to make predictions on a running dataset. For example, such parameters may track false positives and false positives of a classification model. Such parameters may further store false positives and false positives for further processing to improve the accuracy of the model.

[0237] Model updates after deployment may be another aspect of the machine learning cycle. For example, the deployed model may be updated as false positives and / or as false positives are predicted on the production data. In one example, for an MLP model deployed for classification, when a false positive occurs, the deployed MLP model may be updated to increase the probability cutoff for predicting a positive to reduce the false positives. In one example, for an MLP model deployed for classification, when a false negative occurs, the deployed MLP model may be updated to decrease the probability cutoff for predicting a positive to reduce the false negatives. In one example, for an MLP model deployed for classification of surgical complications, when both false positives and false negatives occur, the deployed MLP model may be updated to decrease the probability cutoff for predicting a positive to reduce the false negatives, as predicting a false positive may be less serious than a false negative.

[0238] For example, the deployed model may be updated as more live production data becomes available as training data. In such cases, the deployed model may be further trained, validated, and tested using such additional live production data. In one example, the updated biases and weights of the further trained MLP model may update the biases and weights of the deployed MLP model. Those skilled in the art will recognize that post-deployment model updates may not be a one-time occurrence, but may occur as frequently as is suitable to improve the accuracy of the deployed model.

[0239] ML techniques may be used independently of each other or in combination. Different problems and / or datasets may benefit from using different ML techniques (e.g., combinations of ML techniques). Different training types for models may be more suitable for particular problems and / or datasets. The optimal algorithm (e.g., combination of ML techniques) and / or training type may be determined for a particular use, problem, and / or dataset. For example, processes may be performed to select one or more of the following: select a data reduction type, select a model and / or algorithm configuration, determine the location of data reduction, determine the efficiency of the reduction and / or results, and / or other.

[0240] For example, an ML technique and / or combination of ML techniques may be determined for a particular problem and / or use case. Multiple data reduction and / or data analysis processes may be performed to determine accuracy, efficiency, and / or compatibility associated with a dataset. For example, a first ML technique (e.g., a first set of combined ML techniques) may be used on a dataset to perform data reduction and / or data analysis. The first ML technique may generate a first output. Similarly, a second ML technique (e.g., a second set of combined ML techniques) may be used on a dataset (e.g., the same dataset) to perform data reduction and / or data analysis. The second ML technique may generate a second output. The first output may be compared to the second output to determine which ML technique produced a more desirable result (e.g., a more efficient result, a more accurate result). Multiple ML techniques may be compared on the same dataset to determine the optimal ML technique to use on future similar datasets and / or problems.

[0241] In an example, in a medical context, a surgeon or medical professional may provide feedback to the ML techniques and / or models used on a dataset. The surgeon may input the feedback into the weighted results of the ML model. The feedback may be used as input by the model to determine reduction methods for future analysis.

[0242] In examples, a data analysis method (e.g., an ML technique to be used in the data analysis method) may be determined based on the dataset itself. For example, the origin of the data may influence the type of data analysis method to be used for the dataset. Available system resources may be used to determine the data analysis method to be used for a given dataset. The magnitude of the data may be considered, for example, in determining the data analysis method. For example, the need for an external dataset for local processing levels or magnitude of operational response may be considered (e.g., small device changes may be made using local data, while large device operational changes may require global compilation and validation).

[0243] Such ML techniques may be applied to surgical information (e.g., a combination of the information flows of the surgical information in Figure 7) to generate useful ML models.

[0244] With reference to FIG. 9 , an overview of a surgical system may be provided. Surgical devices or surgical instruments may be used in a surgical procedure as part of the surgical system. A surgical hub / edge device 53000 may be configured to coordinate information flow to the surgical devices or surgical instruments (e.g., a display on the surgical device). For example, a surgical hub / edge device 53000 may be described in U.S. Patent Application Publication No. 2019-0200844(A1), entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Exemplary surgical instruments suitable for use with the surgical system are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety.

[0245] FIG. 9 shows an example of an overview of data flow within a peer-to-peer interconnected surgical system. A surgical hub / edge device 53000 may be used to perform a surgical procedure on a patient in a surgical operating room. A robotic system may be used as part of the surgical system in the surgical procedure. For example, a robotic system may be described in U.S. Patent Application Publication No. 2019-0200844(A1), entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. A robotic hub may be used to process images of the surgical site and then display them to the surgeon through the surgeon's console.

[0246] Other types of robotic systems may be readily adapted for use with the surgical system. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0247] In one example, possibility-based analytics may be deployed to analyze surgical information and / or perform various surgical tasks. Various examples of cloud-based analytics performed by the cloud and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0248] In various aspects, an imaging device may be used in a surgical system and may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0249] The optics of the imaging device may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0250] The one or more illumination sources may be configured to emit electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (e.g., detectable by) the human eye and is sometimes referred to as visible light or simply light. The typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0251] The invisible spectrum (e.g., non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0252] In various aspects, the imaging device may be configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngological-nephroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0253] The imaging device may use multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging can extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is complete to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical equipment is necessary during any surgical procedure. The strict hygiene and sterilization conditions required in an "operating room," i.e., an operating room or procedure room, require the highest possible sterility of all medical devices and equipment. Part of the sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device and its accessories and components. It will be understood that the sterile field may be considered a specific area, such as in a tray or on a sterile towel, that is deemed free of microorganisms, or the sterile field may be considered the area immediately surrounding the patient being prepared for the surgical procedure. The sterile field may include appropriately clothed and hand-washed team members, as well as all equipment and fixtures in the area.

[0254] As shown in FIG. 9, the surgical hub / edge device may be part of the surgical operating room. The operating room may be located within a protected boundary, indicated by the dashed-line enclosure. The protected boundary may be based on (e.g., the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule or Art. 9 General Data Protection Regulation (GDPR). The Privacy Rule may be used to protect health data, which is a special category of personal data and therefore receives a higher level of protection than other personal data.

[0255] In one example, multiple surgical hub / edge devices may be associated with respective operating rooms. A patient 53005 may be undergoing surgery in the operating room. The operating room may include one or more surgical devices (e.g., surgical instruments A53010, B53015, and C53020). The terms surgical device and surgical instrument may be used interchangeably herein. The surgical devices may be used (e.g., autonomously or manually by a medical professional) to perform various tasks associated with a surgical procedure on a patient. How surgical instruments operate autonomously is described in more detail in U.S. Patent Application No. 17 / 747,806, filed May 18, 2022, under the heading "METHOD OF CONTROLLING AUTONOMOUS OPERATIONS IN A SURGICAL SYSTEM," the disclosure of which is incorporated herein by reference in its entirety. For example, the surgical device may be an endocutter. The surgical device may communicate with a surgical hub / edge device 53000 located in the operating room. The surgical hub / edge device 53000 may instruct the surgical device on information related to the surgical procedure being performed on the patient 53005. In an example, the surgical hub / edge device 53000 may set parameters of a surgical instrument (e.g., device) via sending a message to the surgical device, which may be in response to the surgical instrument sending a request message to the surgical hub / edge device 53000 for the parameters. For example, the surgical hub / edge device 53000 may send surgical device information indicating the firing speed of an endocutter that is set on during a stage of the surgical procedure.

[0256] Surgical information associated with the surgical procedure (e.g., surgical data associated with the patient / medical professional / surgical device) may be generated (e.g., by a monitoring subsystem located in the surgical hub / edge device 53000 or locally by the surgical device). For example, the surgical information may be based on the performance of a surgical instrument. For example, the surgical data may be associated with physical measurements, physiological measurements, and / or other measurements. Measurements are described in more detail in U.S. Patent Application No. 17 / 156,28, filed November 10, 2021, under the heading "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0257] Surgical information related to a surgical procedure being performed in an operating room may be transmitted to a local surgical hub / edge device 53000. For example, the operating room may include a surgical display. As the surgical procedure is being performed (e.g., by a medical professional), surgical data (e.g., surgical data associated with measurements obtained from the surgical display) may be transmitted to the surgical hub / edge device 53000, where it may be analyzed. The surgical hub / edge device 53000 may further transmit the surgical information for analysis to an enterprise cloud server (not shown in FIG. 9 ).

[0258] 9 , one or more surgical instruments may be communicatively coupled to the surgical hub / edge device 53000. For example, surgical instrument A 53010, surgical instrument B 53015, and / or surgical instrument C 53020 may be connected to the surgical hub / edge device 53000. The surgical hub / edge device 53000 may perform a discovery operation (e.g., at the start of a surgical procedure or during a transition phase from one surgical step to a subsequent surgical step of a surgical procedure) to discover surgical devices that may be located in the operating room. The surgical instruments may be associated with the surgical procedure being performed.

[0259] The surgical hub / edge device 53000 may, for example, decompose the surgical procedure into surgical tasks or surgical steps based on the surgical procedure. The surgical hub / edge device 53000 may maintain a sequence of surgical tasks or surgical steps in a subsystem or module (e.g., a surgical planning module) located locally to the surgical hub / edge device 53000. The surgical hub / edge device 53000 (e.g., as part of a discovery process) may perform discovery of surgical devices or surgical instruments associated with the surgical procedure and / or surgical steps of the surgical procedure. For example, the surgical hub / edge device 53000 may identify that a colectomy is being performed and that the first step of the colectomy is cutting tissue attached to the colon, thereby mobilizing the colon. Based on this information, the surgical hub / edge device 53000 may send one or more discovery request messages to various surgical devices or surgical instruments to be used during the surgical procedure. The surgical hub / edge device 53000 may receive response messages from various surgical instruments in response to the request message. The response messages from the surgical instruments may include their respective identifications (e.g., which may be referred to as type 53025) and surgical instrument capabilities (e.g., which may be referred to as parameters 53030), as described with respect to FIG. 10 . The surgical hub / edge device 53000 may also receive information indicating the capabilities of the surgical device or surgical instrument in response to the discovery request message. For example, the information may indicate that the surgical instrument is an energy device having a set of surgical instrument capabilities (e.g., standard surgical instrument capabilities). The surgical hub / edge device 53000 may determine that this surgical instrument should be used for the first step of the colectomy and may establish a connection with the surgical instrument. The hub 53000 may receive information in the response message from the instrument indicating that the instrument is an endocutter with standard surgical instrument capabilities. The terms surgical device and surgical instrument may be used interchangeably herein.

[0260] The discovery request message may include an indication that the surgical hub / edge 53000 is requesting information (e.g., characteristics and capabilities) associated with the surgical instrument. In response, the surgical instrument may include the requested information (surgical characteristics and / or surgical parameters 53030) associated with the surgical instrument. For example, the characteristics may include the range of frequencies over which the surgical instrument can operate. The surgical characteristics may include a power rating associated with the surgical instrument. In one example, the surgical hub / edge device 53000 may perform instrument discovery based on a surgical procedure plan associated with the current surgical procedure.

[0261] The surgical hub / edge device 53000 may determine whether to establish a connection with a surgical instrument based on, for example, the characteristics or parameters and the type of surgical instrument. For example, the surgical hub / edge device 53000 may determine that one of the reactive surgical instruments is an endocutter having a frequency operating range that should be used for the anastomosis step of a colectomy. Based on this determination, the surgical hub / edge device 53000 may determine not to establish a connection with the endocutter.

[0262] In one example, as part of the discovery process, the surgical hub / edge device 53000 may assign identification information to the surgical instruments (e.g., each of the surgical instruments) involved in the surgical procedure, and the surgical hub / edge device 53000 may establish a connection with it. For example, after determining whether to establish a connection with the surgical instrument based on the surgical type 53025 and the parameters 53030, the surgical hub / edge device 53000 may assign an identification tag 53035 to the surgical instrument and may transmit the identification tag 53035 to the surgical instrument. As described herein, the identification tag 53035 may be used by the surgical hub / edge device 53000 and / or by the monitoring surgical instrument when requesting data associated with the surgical instrument.

[0263] In one example, the surgical hub / edge device 53000 may determine a role (e.g., monitoring the surgical instrument or a peer surgical instrument being monitored) associated with each of the surgical instruments that are part of the surgical ecosystem. For example, the surgical hub / edge device 53000 may assign one surgical instrument to a monitoring surgical instrument and another surgical instrument to a peer surgical instrument being monitored by the monitoring surgical instrument. The role assignment may include the assignment of respective privileges associated with the surgical instrument, as described with respect to FIG. 10 . For example, if a surgical instrument is determined to be a monitoring surgical instrument, the monitoring surgical instrument may monitor, record, and / or access surgical information (e.g., surgical data) associated with a surgical task being performed on a peer surgical instrument. In one example, a surgical instrument assigned the role of a peer surgical instrument may have the privilege to transmit surgical data to the monitoring surgical instrument. With respect to FIG. 10 , the monitoring surgical instrument and the peer surgical instrument may be connected either directly or through the surgical hub / edge device 53000.

[0264] In one example, a surgical instrument may be pre-configured with a configuration that may enable it to assume the role of a monitoring surgical instrument or a monitored peer surgical instrument. A surgical instrument may be configured and enabled as a monitoring surgical instrument or a peer surgical instrument. In one example, a surgical instrument may determine or select its role based on one or more of the following: the type of peer surgical instrument or the role the peer surgical instrument will play, the surgical instrument capabilities of the peer surgical instrument, the surgical step being performed, the surgical procedure being performed, and / or other. In one example, a surgical instrument may be configured with such information or may request such information from a surgical hub to which the surgical instrument has established a connection. After selecting or enabling a particular role, the surgical instrument may transmit an indication of its selected role to another surgical instrument and / or the surgical hub.

[0265] In one example, if two or more of the surgical instruments indicate that they have assumed the monitoring role, the involved surgical instruments may negotiate to determine which of the surgical instruments should remain in the monitoring role and which of the surgical instruments should change their role to a peer role or have no role. The negotiation may be based on at least the types of surgical instruments involved, the surgical instrument capabilities of the involved surgical instruments, the surgical step being performed, the surgical procedure being performed, and / or other factors. In one example, multiple surgical instruments may both agree to be able to operate in the monitoring role. In one example, a surgical instrument may not have the capability to assume the monitoring role or the capability as a peer. In such a case, the surgical instrument may not be assigned any role and may not be connected to other surgical instruments.

[0266] In one example, negotiation between two surgical instruments may include the transfer of data between the two devices and the application of one or more rules to determine role assignment (e.g., monitoring role vs. monitored role or peer role). The determination may depend on the speed or capabilities of each of the devices, the memory capacity of the devices, timing (e.g., which device sent the discovery request), attributes of the connectivity between the surgical instruments or surgical devices, etc. The determination may be based on whether the type of surgical instrument or type of surgical device is used in a surgical task of the surgical procedure and, optionally, the capabilities of the type of surgical device required for that task or the capabilities of the monitoring surgical instrument (e.g., higher processing speed for processing data, newer model for processing data, larger memory, etc.).

[0267] In one example, a surgical instrument may be powered on during a surgical procedure in a surgical room, for example, after one of the surgical instruments in the surgical room is configured as the monitoring surgical instrument. In such a case, the newly powered surgical instrument may determine that one of the surgical instruments is acting as the monitoring surgical instrument and may then assume its role as a peer surgical instrument and establish a connection with the existing monitoring surgical instrument. In one example, the existing monitoring surgical instrument may indicate its status as a monitoring surgical instrument to the newly added surgical instrument.

[0268] In one example, once a surgical instrument assumes its role as a monitoring surgical instrument, it may then have the ability to directly monitor performance and pull data directly from the surgical instrument without using the surgical hub / edge device 53000. In one example, the monitoring surgical instrument may request information about peer surgical instruments from the surgical hub / edge device 53000. For example, as shown in FIG. 9 , surgical instrument A 53010 may be configured as a monitoring surgical instrument (e.g., based on the surgical instrument capabilities of surgical instrument A 53010). Assuming surgical instruments B 53015 and C 53020 are configured as or assume the role of peer surgical instruments, surgical instrument A 53010 may be able to directly monitor and / or record surgical information and / or performance of surgical instruments B 53015 and C 53020.

[0269] In one example, surgical instrument A53010 may monitor (at surgical instrument B53015 and / or C53020) surgical data associated with surgical steps of a surgical procedure. In one example, surgical instrument A53010 may request surgical information or surgical data associated with the execution of surgical tasks being performed on each of surgical instruments B53015 and C53020 directly from surgical instruments B53015 and C53020 (e.g., send a message requesting the data) without involving the surgical hub / edge device 53000. In one example, surgical instrument A53010 may request data associated with the execution of surgical tasks being performed on each of surgical instruments B53015 and C53020 from or via the surgical hub / edge device 53000. As described with respect to FIG. 10, the surgical hub / edge device 53000 may determine whether the monitoring surgical instrument can monitor the surgical instrument directly or indirectly, for example, via the surgical hub / edge device 53000.

[0270] Monitoring a surgical device or surgical instrument may include a monitoring surgical device (e.g., its own monitoring surgical instrument or a monitoring surgical instrument in cooperation with the surgical hub / edge device 53000) collecting surgical information associated with a patient, a healthcare provider, and / or a surgical task being performed by the surgical instrument being monitored. The surgical information associated with the patient, healthcare professional may include measurements related to a physical condition, a physiological condition, and / or other factors. The surgical information associated with a surgical instrument may include performance metrics associated with the surgical instrument or the task being performed by the surgical instrument.

[0271] Determining whether the monitoring surgical instrument may directly interact with a peer surgical instrument may be determined by a machine learning model 53040 located on the surgical hub / edge device 53000, as described herein in Figure 12. The machine learning model 53040 may be trained to consider the type 53025 of the peer surgical instrument, the surgical instrument capabilities of the peer surgical instrument, the surgical step being performed, and / or the surgical procedure being performed when determining whether the monitoring surgical instrument may directly interact with a peer surgical instrument.

[0272] In one example, the monitoring surgical instrument may receive (e.g., from the surgical hub / edge device 53000) a list of potential peer surgical instruments that it may monitor. The monitoring surgical instrument may also receive instructions identifying peer surgical instruments that the monitoring surgical instrument may be able to monitor directly and peer surgical instruments that the monitoring surgical instrument may monitor in cooperation with the surgical hub / edge device 53000.

[0273] In one example, a monitoring surgical instrument may receive instructions to monitor a set of peer surgical instruments. The instructions may include a list of identification tags 53035 associated with the peer surgical instruments. The monitoring surgical instrument may store locally (e.g., in local memory) the list of peer surgical instruments to be monitored.

[0274] In one example, the surgical hub / edge device 53000 may obtain a list of surgical instruments that may be utilized during a surgery. As part of the surgical procedure, for example, the surgical hub / edge device 53000 may assign roles to be assigned to the surgical instruments. The surgical hub / edge device 53000 may communicate the roles to the devices involved, for example, by sending messages to the surgical instruments.

[0275] In one example, the surgical hub / edge device 53000 may update the role and / or privileges assigned to the surgical instrument. For example, the role may be updated during a transition from one surgical step of a surgical procedure to another surgical step of the surgical procedure. In one example, a surgical instrument that may have previously been assigned a monitoring role may be updated to a peer surgical instrument and may be monitored by another surgical instrument, for example, a newly powered surgical instrument. The surgical hub / edge device 53000 may send an update message to the surgical instrument indicating that the surgical instrument is changing its role from a monitoring surgical instrument to a peer surgical instrument. The surgical hub / edge device 53000 may also indicate to the surgical instrument the identity of the new monitoring surgical instrument.

[0276] In one example, the surgical instrument A53010 may receive surgical information directly from the surgical instrument C53020. The surgical instrument A53010 may receive surgical information periodically or aperiodically (e.g., based on the completion of a surgical task in the surgical instrument B53015 or C53020, or based on a trigger condition being met (e.g., starting and / or finishing a particular instrument action such as clamping, firing, etc., or when a derived parameter is outside an expected range / threshold)). For example, the surgical instrument A53010 may request and / or receive surgical parameters related to tissue it may be dissecting or mobilizing. In the exemplary surgical instrument, A53010 may request and / or receive surgical information associated with the surgical tasks of the surgical procedure from the surgical instrument B53015 indirectly via the surgical hub / edge device 53000.

[0277] In one example, the machine learning model and / or trained machine learning model may be utilized as part of a supervised learning framework, as described with respect to FIG. 9 . Supervised learning models are described herein in FIG. 8A . Training data (e.g., training examples 802 as illustrated in FIG. 8A ) may consist of a set of training examples (e.g., input data mapped to labeled outputs, as shown in FIG. 8A ). The training data used in training the local machine learning model 53040 may include surgical data collected from previous surgical procedures and / or simulated surgical procedures. The training data may include attributes or parameters associated with a patient and / or parameters associated with a surgical instrument. In one example, the machine learning model as an output may provide parameters associated with another surgical instrument. For example, a machine learning model may be utilized to identify the size and color of a cartridge to be used for a smart stapling device as an output. As inputs, the machine learning model may be provided with various parameters collected by the surgical instrument (e.g., power, time, impedance values ​​collected by the energy device) and parameters associated with the patient (e.g., tissue thickness measured by the jaws of the surgical instrument, incision area, patient age, etc.) The machine learning model, based at least on the surgical instrument parameters collected by the surgical instrument and the parameters associated with the patient, may predict the size and color of a cartridge to be used by the surgical stapling device.

[0278] In one example, a local machine learning model 53040 located in the surgical hub / edge server device 53000 may use surgical information and surgical parameters associated with the patient, medical professional, and / or surgical instrument to predict surgical instrument settings or identify surgical instrument parts (e.g., cartridges) as a result. The surgical hub / edge device 53000 may transmit the predicted results to the monitoring surgical instrument.

[0279] In one example, a local machine learning model may reside within a peer surgical instrument, as described herein in FIG. 12. The local machine learning model within the peer surgical instrument may result in a prediction of cartridge size and color based on surgical instrument parameters and / or patient parameters. The peer surgical instrument may transmit the results to the monitoring surgical instrument for use.

[0280] In one example, the local machine learning model may reside within the supervising surgical instrument, as described herein in FIG. 12. In such cases, the peer surgical instrument may directly or indirectly transmit surgical information and parameters associated with the patient, medical professional, or surgical task being performed by the peer surgical instrument to the supervising surgical instrument. The local machine learning model located within the supervising surgical instrument may predict surgical instrument settings or identify surgical instrument parts (e.g., cartridges) as a result. The supervising surgical instrument may use the predicted results, including surgical instrument settings and / or surgical instrument part selection.

[0281] In one example, the surgical procedure to be performed may be a colectomy. During the anastomosis step of the surgical procedure, the endocutter may be configured or configured to be itself the monitoring surgical instrument, and the energy device may be configured to be a peer surgical instrument monitored by the monitoring surgical instrument. The energy device (which is the surgical instrument being monitored) may transmit surgical information to the endocutter (the monitoring device). The surgical information may include information about the anatomical structure of the tissue being monitored, such as tissue thickness. In one example, the energy device may transmit the surgical data based on a request received from the endocutter. In one example, the energy device may transmit the surgical information to the endocutter based on a trigger condition being met, as described herein. In one example, the surgical information may be transmitted to the endocutter periodically (e.g., based on a timer configured on the energy device). The endocutter may store the surgical data and perform analysis on the surgical data, as described herein. In examples, a monitoring surgical instrument (e.g., an endocutter) may provide recommendations to a monitored surgical instrument (e.g., an energy device) to adjust one or more of its parameters (e.g., surgical instrument parameters) based on an analysis of the tissue thickness. For example, the endocutter may analyze the tissue thickness and determine the uniqueness of the tissue thickness. Based on this analysis, the endocutter may send recommendations (e.g., updated recommendations) to the energy device to set its power settings accordingly, for example, when performing a surgical task of the surgical procedure.

[0282] The analysis performed within the endocutter may include a machine learning model 53040, which may take data (e.g., measurements) from the energy device as input and output recommendations for setting one or more instrument parameters. The endocutter may perform or send recommendations to a third device assisting in performing the surgical step at hand based on surgical data (e.g., surgical measurements) received from the energy device. For example, measurements from the energy device may be received by the endocutter indicating that the patient's tissue thickness is greater than average. Based on this, the endocutter may send a message to a third device, such as a robotic arm or clamp, to reorient itself to a different position (e.g., based on the greater tissue thickness of the tissue), which may allow the energy device to move more freely within the surgical site. The endocutter may send recommendations to a device performing or assisting in performing a surgical task (e.g., a future surgical task) based on surgical information (e.g., surgical measurements) received from the energy device.

[0283] FIG. 10 shows a message sequence diagram illustrating one surgical instrument (e.g., surgical instrument A 53050) working in conjunction with a surgical hub / edge device 53045 to monitor other surgical instruments (e.g., surgical instrument B 53055 and surgical instrument C 53060).

[0284] In one example, the surgical hub / edge device 53045 may statically obtain a list of surgical instruments present in the operating room, and information regarding their respective surgical instrument types and / or surgical instrument capabilities, from a surgical instrument list associated with the surgical procedure plan or surgical procedure (e.g., a list of surgical instruments that have been activated and will be used in the surgical procedure).

[0285] 10, a surgical hub / edge device 53045 (e.g., a local surgical hub / edge device) may dynamically obtain the surgical instruments involved in a surgical procedure by initiating a discovery procedure. For example, at 53070, the surgical hub / edge device 53045 may send a discovery message to surgical instruments A / B / C / D in the operating room where the surgical procedure is being performed. The surgical hub / edge device 53045 may be configured (e.g., pre-configured) with a list of surgical instruments and timestamps for when the surgical instruments may be powered on and available for communication.

[0286] At 53072, each of the surgical instruments may determine its surgical instrument type and surgical instrument capabilities. In one example, the surgical instruments may be configured (e.g., pre-configured) with a set of surgical instrument types and surgical instrument capabilities. At 53072, each of the surgical instruments may generate a surgical instrument type and a surgical instrument capability.

[0287] At 53075, each of the surgical instruments may send a response message 53075 to the surgical hub / edge server 53045 in response to the discovery request message 53070. The response message 53075 may include an indication of the surgical instrument type and surgical instrument capabilities associated with the surgical instrument sending the response message. The surgical instrument capabilities may include qualities related to the performance and / or intelligence of the surgical instrument. Qualities related to the performance and / or intelligence of surgical instruments are described in more detail in U.S. Patent Application No. 17 / 156,28, filed November 10, 2021, under the heading "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0288] The surgical hub / edge device 53045 may assign a role to an available surgical instrument B or C, for example, based on the response message 53075 from the surgical instrument (e.g., each of the surgical instruments). A surgical instrument may be assigned a role as a monitoring surgical instrument (e.g., surgical instrument A) or a peer surgical instrument (e.g., surgical instrument B or C) being monitored by the monitoring surgical instrument.

[0289] In one example, a surgical instrument (e.g., surgical instrument D 53065) may not be assigned the role of a monitoring or peer surgical instrument based on its surgical instrument type and / or surgical instrument capability information. For example, surgical instrument D 53065 may lack the capability to establish a point-to-point connection with another surgical instrument. In one example, the surgical hub / edge device 53045, after receiving a response from surgical instrument D 53065, may determine that the surgical instrument's capabilities (e.g., operating power) are not within an acceptable operating range and therefore may not be assigned a monitoring or peer role.

[0290] In one example, based on the capabilities of the surgical instrument, the surgical hub / edge device 53045 may determine that the surgical instrument (e.g., surgical instrument A 53050) is a smart surgical instrument and may therefore be assigned the role of a monitoring surgical instrument. Based at least on determining that the surgical instrument is a smart surgical instrument (e.g., has sufficient processing and memory capabilities to perform monitoring and recording of surgical tasks being performed by peer surgical instruments), the surgical hub / edge device 53045 may determine and / or assign the surgical instrument as the role of a monitoring surgical instrument.

[0291] At 53080, the surgical hub / edge device 53045 may send an assignment message to the surgical instrument 53050 indicating that it has been assigned the role of a monitoring surgical instrument. In the assignment message, the surgical hub / edge device 53045 may include an indication that surgical instrument A 53050 can establish a direct peer-to-peer connection with surgical instrument B 53055. The surgical hub / edge device 53045 may send another assignment message 53082 to surgical instruments B 53055 and C 53060 indicating that each surgical instrument has been assigned the role of a peer surgical instrument. The assignment message 53082 may instruct surgical instrument B 53055 to establish a direct peer-to-peer connection with surgical instrument A 53050. The assignment message 53082 may instruct surgical instrument C 53060 to establish a direct peer-to-peer connection with surgical instrument A 53050.

[0292] 10 , privileges associated with the assigned role may be included in the assignment message. For example, surgical instrument A 53050 assigned as the monitoring surgical instrument may be assigned read and write privileges with respect to the peer data of surgical instrument B. Surgical instrument A 53050 may record data of surgical instrument B 53055 in the local memory of surgical instrument A. Privileges for monitoring surgical instrument A 53050 may include sending commands and / or recommendations to the peer surgical instrument 53085 as related to the performance of surgical steps. Surgical instrument B 53055 assigned as the peer surgical instrument 53085 may be assigned privileges to send surgical information to the monitoring surgical instrument.

[0293] In one example, the local surgical hub / edge device 53045 may indicate that the monitoring surgical instrument may indirectly connect to the peer surgical instrument, e.g., the monitoring surgical instrument may access the peer surgical instrument's data via the surgical hub / edge device 53045. As described with respect to FIG. 10 , the surgical hub / edge device 53045 may indicate to surgical instrument A 53050 to establish a connection with surgical instrument C 53060 indirectly via the surgical hub / edge device 53045, which may be based on the surgical instrument capabilities of surgical instrument C 53060. This message may also be sent to surgical instrument C 53060.

[0294] At 53084, the monitoring surgical instrument 53050 may establish a peer-to-peer connection with peer surgical instrument B 53055 and surgical instrument C 53060. The established peer-to-peer connection may be utilized to monitor and / or record surgical information associated with the surgical task being performed on the peer surgical instrument 53055.

[0295] In one example, a monitoring surgical instrument may establish a connection with a peer surgical instrument at the start of a surgical procedure. For example, if a surgical procedure includes surgical steps 1 through K, peer-to-peer connection establishment may occur as part of surgical step 1.

[0296] In one example, the role assigned to a surgical instrument may change during the transition from one surgical step to a subsequent surgical step. For example, during the transition from surgical step 1 to surgical step 2 of a surgical procedure, the assigned role of surgical instrument A53050 may change from a supervisory surgical instrument to a peer surgical instrument. In such a case, during surgical step 2, the surgical instrument A53050 with the newly assigned role may no longer have the privileges of a supervisory surgical instrument.

[0297] As the surgical instruments perform their respective surgical tasks associated with the surgical steps, they may generate surgical information regarding how they are performing those surgical tasks. This surgical data may be transmitted to or accessed by the monitoring surgical instrument 53050, either directly without involving the surgical hub / edge device 53045, or indirectly through the surgical hub / edge device 53045.

[0298] At 53091, the peer surgical instruments B 53055 and C 53060 may generate surgical information associated with a patient, a medical professional, or a surgical task performed by the surgical instrument. At 53092, the peer surgical instrument B 53055 may transmit the surgical information to the monitoring surgical instrument A 53050, for example, using the peer-to-peer connection established at 53084. At 53093, the peer surgical instrument C 53060 may transmit the surgical information to the monitoring surgical instrument A 53050, for example, using the peer-to-peer connection established at 53084. The surgical information transfer between the monitoring surgical instrument A 53050 and the peer surgical instruments B 53055 and / or C 53060 may be performed under the supervision of the surgical hub / edge device 53045.

[0299] 11a shows an exemplary message sequence diagram for establishing a peer-to-peer connection with one or more surgical hub / edge devices 53100 and surgical instruments (e.g., surgical instruments A 53095, B 53105, C 53110, and D 53115) without involving any centralized surgical computing device. Surgical instrument A 53095 may, as part of the initiation of a surgical procedure, obtain information (e.g., capability information) regarding other surgical instruments B / C / D and surgical hubs that may be active in the surgical operating room and / or connected to the ecosystem. The smart surgical instrument may identify that other surgical instruments and surgical hubs are connected to the ecosystem and determine that the other surgical instruments and surgical hubs can establish a peer-to-peer connection during the surgical procedure.

[0300] 11 , at 53117, a surgical instrument (e.g., surgical instrument A53095) may determine, for example, based on its capabilities, whether it can assume a surgical instrument monitoring role as described herein. For example, surgical instrument A53095 may determine that it is a smart surgical instrument (e.g., a smart surgical stapler) and / or that it is the only or one of the smart surgical instruments to be utilized in a surgical procedure. In one example, surgical instrument A53095 may determine that it is operating within an interconnected network that can monitor other surgical instruments (e.g., surgical instrument B53105 or C53110) by establishing a peer-to-peer connection with those surgical instruments.

[0301] In one example, the surgical instrument A53095 may be a smart surgical instrument. For example, the surgical instrument may operate independently, identify surgical instruments other than itself, and determine that it can communicate with the identified surgical instrument over a network. The network may be a local area network (LAN), a wireless interface (e.g., a WiFi interface (WiFi 6, WiFi 6E, etc.), a Bluetooth X interface, etc.), and / or an optical interface (e.g., an optical fiber-based LAN). Devices in the network may include a smart computing device (e.g., a smart surgical hub) or server (e.g., an edge server) at the center of the network. The network may be located within a secure perimeter (e.g., a HIPAA perimeter).

[0302] In one example, a surgical instrument may identify and / or monitor other devices without utilizing a centralized computing device. In such a configuration, surgical information (e.g., surgical information associated with surgical tasks) may be exchanged directly between smart surgical instruments without utilizing a central surgical computing device or server. In one example, a surgical instrument may determine that it has the capability to be a monitoring device, i.e., the capability to monitor and / or record surgical information associated with one or more surgical tasks being performed on other surgical instruments (e.g., other peer surgical instruments). In one example, the surgical instrument may be capable of monitoring communications between two smart devices and recording aspects of their interactions or streams to monitor their operation. In one example, the surgical instrument may be capable of monitoring its own operation. Based at least on these determinations, surgical instrument A 53095 may configure itself as a monitoring surgical instrument.

[0303] In one example, surgical instrument A 53095 may analyze surgical instrument capability information it may receive from a set of peer surgical instruments (e.g., surgical instrument B 53105 and surgical instrument C 53110). Based on its analysis of the surgical instrument capability information associated with the set of peer surgical instruments (e.g., limitations of the peer surgical instruments), surgical instrument A 53095 may determine that it is the only or one of the smart surgical instruments that should be utilized during the surgical procedure. Thus, surgical instrument A may configure itself as a monitoring surgical instrument.

[0304] In one example, one of the smart surgical instruments utilized in a surgical procedure may determine that multiple other smart surgical instruments are also utilized in the surgical procedure. The smart surgical instrument may, for example, as part of a discovery procedure, obtain the firmware / software version (e.g., the version of the ML software) running on each of the smart surgical instruments utilized in the surgical procedure. The smart surgical instrument may compare its firmware / software version with the firmware / software versions of the other surgical instruments and determine that it is running the most recent version of the firmware / software. Based on this determination, the smart surgical instrument may configure itself as a monitoring surgical instrument.

[0305] The surgical instrument A53095 may initiate a discovery procedure. The surgical instrument A53095 may obtain (e.g., from a pre-configuration or obtained from the surgical hub / edge device 53100) a list of surgical instruments that may be utilized during the surgical procedure. At 53120, the instrument A53095 may send a discovery message to one or more surgical instruments and / or surgical hub / edge devices that may, for example, be part of the surgical procedure.

[0306] In one example, the surgical hub / edge device 53100 may assign roles to the surgical instruments (e.g., as described with respect to FIG. 10 ). After roles are assigned, the monitoring surgical instrument may control and perform other actions described herein. For example, the monitoring surgical instrument may send discovery requests to determine with which of the surgical instruments it can directly establish a connection.

[0307] At 53122, the surgical instruments may determine their respective surgical instrument types and surgical instrument capabilities. In one example, the surgical instruments may be configured (e.g., pre-configured) with a set of surgical instrument types and / or surgical instrument capabilities. The surgical instrument types and surgical instrument capabilities may be stored in local memory of the surgical instruments.

[0308] At 53125, each of the surgical instruments and surgical hubs that received the discovery message from the monitoring surgical device may respond with a response message. The response message sent by each of the surgical instruments or received by the monitoring surgical device may include an indication of the surgical instrument type and surgical instrument capabilities, for example, as determined at 53122. The surgical instrument capabilities may include qualities related to the performance and / or intelligence of the surgical instrument, which may be described in more detail in U.S. Patent Application No. 17 / 156,28, filed November 10, 2021, under the heading "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0309] The monitoring surgical instrument (e.g., surgical instrument A 53095) may assign peer surgical instrument roles to available surgical instruments A / B / C / D and / or the surgical hub / edge device 53100, for example, based on response messages from the surgical instruments. Peer surgical instrument role assignment may be based on selection criteria, which may include surgical instrument type, surgical instrument capabilities, surgical step of the surgical procedure, surgical procedure, etc.

[0310] In one example, a surgical instrument (e.g., surgical instrument D 53115) may not be assigned the role of a peer surgical instrument based on its surgical instrument type and / or surgical instrument capability information. For example, surgical instrument D 53065 may lack the capability to establish a point-to-point connection with another surgical instrument. In one example, the surgical monitoring instrument 53095, after receiving a response from surgical instrument D 53115, may determine that the surgical instrument's capabilities (e.g., operating power) are not within an acceptable operating range and therefore may not be assigned the peer role.

[0311] At 53130, the monitoring surgical instrument A 53095 may send an assignment message to each of surgical instrument B / C and surgical hub / edge device 53100 indicating that each of surgical instrument B / C and surgical hub / edge device 53100 has been assigned the role of a peer surgical instrument. In one example, the assignment message may include privileges associated with the peer role assigned to the surgical instrument and / or surgical hub. For example, the monitoring surgical instrument may assign surgical instrument B 53105 and surgical instrument C 53110 as peer surgical instruments.

[0312] In one example, the assignment message, the surgical monitoring surgical instrument A53095, may include an indication that the surgical instrument may establish a peer-to-peer connection with the surgical instrument A53095. In one example, as part of establishing the peer-to-peer connection, the surgical instrument A53095 and the peer-to-peer surgical instrument may optimize various parameters of the peer-to-peer connection (e.g., surgical data sharing, data transfer rate, etc.).

[0313] At 53131, the monitoring surgical instrument A53095 may establish a peer-to-peer connection with the surgical computing device / edge server 53100. The established peer-to-peer connection may be used to monitor and / or record surgical information on the surgical computing device / edge server 53100.

[0314] At 53132, the monitoring surgical instrument A 53095 may establish a peer-to-peer connection with the peer surgical instrument B 53105. The established peer-to-peer connection may be utilized to monitor and / or record surgical information on the peer surgical instrument B 53105.

[0315] At 53133, the monitoring surgical instrument 53095 may establish a peer-to-peer connection with the peer surgical instrument C53110. The established peer-to-peer connection may be utilized to monitor and / or record surgical information on the peer surgical instrument C53110.

[0316] In one example, the monitoring surgical instrument A53095 may establish a direct peer-to-peer connection with a peer surgical instrument at the start of a surgical procedure. For example, if the surgical procedure includes surgical steps 1 through K, the peer-to-peer connection establishment may occur as part of surgical step 1.

[0317] At 53126, the peer surgical instruments B53105 and C53110 may generate surgical information associated with the patient, medical professional, or surgical instrument. At 53127, the surgical instrument may transmit the surgical information to the monitoring surgical instrument A53095.

[0318] In one example, a monitoring surgical instrument, e.g., a smart surgical stapling device, may identify a surgical energy device to be used during a surgical procedure in an operating room. The smart surgical stapling device may search for the capabilities of the surgical energy device and configure it as a peer surgical instrument to be monitored by the smart energy stapler. The smart surgical stapling device may establish a peer-to-peer connection with the surgical energy device. As part of the surgical task, the surgical energy device may be used to incise and / or mobilize tissue. During this surgical task, the energy device may record and / or process tissue viability, for example, based on feedback of various surgical parameters collected by the surgical energy device. The surgical parameters may include power, time, impedance, etc. The smart surgical stapling device may directly obtain information collected by the energy device (e.g., parameters including power, time, and impedance) via the established peer-to-peer connection. In one example, the energy device may calculate surgical instrument settings, such as an initial starting speed of a motor for firing staples, and send them to the smart surgical stapling device. In one example, based on information obtained directly from the energy device, the smart surgical stapling device may calculate an initial starting speed of a motor for firing staples. In one example, based at least on information obtained directly from the energy device, the smart surgical stapler may identify an optimal location for tissue incision with the stapling device. The location for tissue incision may be based on tissue characteristics / disease state of the tissue or area with minimal vascular avoidance. In one example, based at least on the area incised, the energy device may identify a cartridge (e.g., cartridge size (45 mm or 60 mm) and / or cartridge color (e.g., blue) based on the thickness of the tissue collected in the jaws). The energy device may communicate cartridge identification information directly to the smart energy device using a peer-to-peer connection between the energy device and the smart stamp device.

[0319] In one example, interconnections may change when transitioning from one surgical step to a subsequent surgical step. For example, interconnections and privilege assignments may be adjusted from surgical step 1 to surgical step 2. For example, with reference to FIG. 11 , during the transition from surgical step 1 to surgical step 2, monitoring surgical instrument A 53095 may determine that surgical instrument B 53105 may no longer be a peer surgical instrument.

[0320] In one example, as surgical instruments perform their respective surgical tasks associated with a surgical step, they may generate surgical data related to how they are performing those surgical tasks, which may be described in more detail in U.S. patent application Ser. No. 17 / 156,28, filed November 10, 2021, under the heading "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety. This surgical data may be accessed by the monitoring surgical instrument either directly, as described herein with respect to FIG. 11, or indirectly via the surgical hub / edge device 53100, as described herein with respect to FIG. 10.

[0321] FIG. 12 illustrates an example of the relationship between a surgical computing device (e.g., as a surgical hub / edge device) or monitoring surgical device 53135 and a surgical instrument 53140. Surgical information (e.g., surgical data) may be transmitted from the surgical computing device or monitoring surgical device 53135 to the surgical instrument 53140, or vice versa. In an example, the surgical information may be communicated through a network interface 53145. The network interface may be of many types, as described herein. The surgical information may include surgical data associated with a surgical task being performed on the surgical instrument 53140. The surgical data may include data based on measurements obtained from sensors, actuators, robotic movements, biomarkers, surgeon's biomarkers, visual aids, and / or others. The measurements are described in more detail in U.S. Patent Application No. 17 / 156,28, filed November 10, 2021, under the title "Monitoring Of Adjusting A Surgical Parameter Based On Biomarker Measurements," the disclosure of which is incorporated herein by reference in its entirety.

[0322] The surgical information or surgical data measurements may be associated with one of many actuators located in the operating room. For example, the surgical information may be generated from measurements of potentiometer readings. This surgical information may be associated with the orientation of the surgical instrument. The surgical information may be used in assessing how the surgical instrument is performing its respective surgical task, as described with respect to FIGS. 9 and 10. The surgical information may be used in determining the role of the surgical instrument, as described with respect to FIGS. 10 and 11.

[0323] 12 , the surgical computing device or overseeing surgical device 53135 may include, among other things, a processor 53137, memory 53139 (e.g., non-removable and / or removable memory), a machine learning model 53143, and / or a local storage subsystem 53144. It will be understood that the surgical computing device or overseeing surgical instrument 53135 may include any sub-combination of the foregoing elements / subsystems while remaining consistent with an embodiment.

[0324] The processor 53137 in the surgical computing device or monitoring surgical device 53136 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 53137 may perform data processing, authentication, input / output processing, and / or any other function that may enable the surgical computing device or monitoring surgical device 53136 to operate in an environment suitable for performing a surgical procedure. The processor 53137 may be coupled to a transceiver (not shown). The processor 53137 may use a transceiver (not shown in the drawings) to communicate with a peer surgical instrument 53140.

[0325] The memory 53139 in the surgical computing device or monitoring surgical instrument 53135 may be used to store the location where the surgical information was sent. For example, the memory may be used to recall that the surgical information was sent to the peer surgical instrument 53140. The memory may include a database and / or lookup table. The memory may include virtual memory that may be linked to a server located within the protected network.

[0326] The processor 53137 in the surgical computing device or monitoring surgical instrument 53135 may access information from and store data in any type of suitable memory (e.g., non-removable and / or removable memory). Non-removable memory may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory may include secure digital memory.

[0327] The processor 53137 in the surgical computing device or monitoring surgical device 53135 may access information from and store data in the expansion storage 53144 (e.g., non-removable memory and / or removable memory). In one example, the processor 53137 may access information from and store data in memory that is not physically located on the surgical computing device or monitoring surgical device 53135, such as on a server or secondary edge computing system (not shown).

[0328] The processor 53137 in the surgical computing device or monitoring surgical device 53135 may utilize a machine learning model 53143, as described herein, to predict parameters associated with the surgical instrument or identify a portion of the surgical instrument (e.g., a stapler cartridge). The processor 53137 may use a transceiver (not shown in the drawings) to communicate the surgical information or predicted surgical parameters or predicted identification of the surgical portion directly to the peer surgical instrument 53140. Direct communication between the surgical computing device or monitoring surgical instrument 53135 and the peer surgical instrument 53140 may occur using an established peer-to-peer connection 53145.

[0329] 12 , the peer surgical instrument 53140 may include, among other things, a processor 53136, a memory 53138 (e.g., non-removable memory and / or removable memory), a local machine learning model 53145, and / or a local storage subsystem 53148. The local machine learning model may be simpler than the machine learning model used in the surgical computing device or the surgical monitoring device 53135. In one example, the local machine learning model may comprise a training model that may be utilized, for example, to predict parameters associated with the surgical instrument or to identify portions of the surgical instrument. It will be understood that the computing device 53140 may include any sub-combination of the foregoing elements / subsystems while remaining consistent with an embodiment.

[0330] The processor 53136 in the peer surgical instrument 53140 may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 53136 may perform data processing, authentication, input / output processing, and / or any other function that may enable the peer surgical instrument 53140 to operate in an environment suitable for performing a surgical procedure. The processor 53136 may be coupled to a transceiver (not shown). The processor 53136 may communicate with the surgical computing device or the monitoring surgical instrument 53135 using the transceiver (not shown in the drawings).

[0331] The memory 53138 in the peer surgical instrument 53140 may be used to store the location to which the surgical information was sent. For example, the memory may be used to recall that surgical information was sent to the peer surgical instrument 53140. The memory may include a database and / or lookup table. The memory may include virtual memory, which may be linked to a server located within the protected network.

[0332] The processor 53136 in the peer surgical instrument 53140 may access information from and store data in any type of suitable memory (e.g., non-removable and / or removable memory). Non-removable memory may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory may include secure digital memory.

[0333] The processor 53136 in the peer surgical instrument 53140 may access information from and store data in the extended storage 53148 (e.g., non-removable memory and / or removable memory). In one example, the processor 53136 may access information from and store data in memory that is not physically located on the peer surgical instrument 53140, such as on a server or a secondary edge computing system (not shown).

[0334] The processor 53136 in the peer surgical instrument 53140 may utilize a local machine learning model 53145, as described herein, to predict parameters associated with the surgical instrument or identify a portion of the surgical instrument (e.g., a stapler cartridge). The processor 53136 may use a transceiver (not shown in the drawings) to communicate the surgical information or predicted surgical parameters or predicted identification of the surgical portion directly to the monitoring surgical instrument 53135. Direct communication between the peer surgical instrument 53140 and the surgical computing device or monitoring surgical instrument 53135 may occur using a peer-to-peer connection established through the interface 53145.

[0335] FIG. 13 illustrates peer-to-peer interconnected surgical instruments or surgical devices without using a central surgical hub for remote monitoring / recording. At 53150, a surgical device or surgical instrument may determine that it has the capability to monitor and record surgical data associated with a surgical task of a surgical procedure being performed on a second surgical instrument. The capability of a surgical instrument to be a monitoring surgical instrument may include the monitoring surgical instrument having the ability to access surgical data from the second surgical instrument and / or the ability to set (e.g., remotely set) parameters on the second surgical instrument based on the accessed surgical data. The surgical data may include surgical data associated with a patient, a medical professional, or a surgical instrument. Based on the determination, the surgical instrument may configure itself as a monitoring surgical instrument. In one example, the surgical instrument being a monitoring surgical instrument and the second surgical instrument being a peer surgical instrument being monitored by the monitoring surgical instrument is based on negotiation between the surgical instrument and the second surgical instrument.

[0336] At 53152, the surgical instrument may determine that it has the capability to be a peer surgical instrument by which a second surgical instrument may be monitored. The capability to be a peer surgical instrument may include having the capability to establish a peer-to-peer connection with the monitoring surgical instrument and / or having the capability to collect surgical data associated with a patient, a medical professional, or a surgical instrument and transmit the collected surgical information to the monitoring surgical instrument. The surgical instrument may configure the second surgical instrument as a peer surgical instrument.

[0337] At 53154, the surgical instrument may establish a peer-to-peer connection with a second surgical instrument. The peer-to-peer connection is established between the first surgical instrument and the second surgical instrument for the first surgical instrument to monitor and record surgical information of a surgical task on the second surgical instrument.

[0338] At 53156, the surgical instrument may begin monitoring and recording surgical data associated with the second surgical instrument using the established peer-to-peer connection with the second surgical instrument.

[0339] 14 illustrates a discovery mechanism used to assign roles (e.g., a monitoring role and / or a peer role) to surgical instruments that may be utilized in a surgical procedure. At 53158, a first surgical instrument may send a discovery request indication to a second set of instruments associated with the surgical procedure.

[0340] At 53159, the first surgical instrument may receive an indication in the reply message from each of the set of second surgical instruments. The indication in the reply message may include an indication of a surgical instrument type and an indication of a capability of each of the second surgical instruments. Based on the surgical instrument type and the surgical instrument capabilities, the first surgical instrument may determine that each of the second surgical instruments is a peer surgical instrument. The indication in the reply message from the first surgical instrument to each of the set of second surgical instruments may indicate an assigned role.

[0341] At 53160, based at least on the surgical instrument type and capabilities of each of the second surgical instruments, the first surgical instrument may determine that each of the set of second surgical instruments is a peer surgical instrument.

[0342] The first surgical instrument may be capable of monitoring one of the second surgical instruments. The first surgical instrument may be a monitoring surgical instrument and may be able to access data of one of the second surgical instruments that has been assigned the role of a peer surgical instrument. In one example, the first surgical instrument may be able to set parameters of the second surgical instrument based on the accessed surgical data.

[0343] In one example, the roles of the first surgical instrument and the second surgical instrument may be determined based on a negotiation between each of the first and second surgical instruments.

[0344] In one example, a first surgical instrument may assume the role of a supervisory surgical instrument based at least on its own surgical instrument type and capability information.

[0345] In one example, the first surgical instrument may be a smart surgical instrument (e.g., operating within an interconnected network may be able to understand the limitations of a second surgical instrument used in the surgical procedure). This may include the instrument recognizing that it is the only smart instrument in the procedure as well as identifying that other instruments have surgical instrument capabilities to share data.

[0346] In one example, a set of surgical instruments may be utilized in performing a surgical procedure. Some of the surgical instruments, e.g., smart stapling devices, may be smarter and / or more advanced than, for example, energy devices. The advancement of the smart stapling devices over the energy devices may be based on revisions or levels of software (e.g., machine learning software) installed on each of the surgical devices.

[0347] In one example, during activation of a procedure, the smart surgical stapler may obtain information about other surgical instruments that may be active and / or interconnected to the ecosystem. The smart surgical stapler may have the availability of other devices identified based on the available instruments and confirmation of which operations it is capable of performing during the procedure based on the instruments in the operating room. Based on the identification of the identified available instruments, the smart surgical stapler may attempt to directly connect to the other instruments to have a peer-to-peer connection that may optimize data sharing, transfer speed, and / or other factors. For example, an energy device may be used to incise and mobilize tissue. During the process, tissue viability may be recorded / processed based on feedback of parameters (e.g., power, time, impedance, etc.) collected by the energy device. Information collected from the energy device may be communicated to the surgical stapler to indicate the initial starting speed of the motor for firing the staples. This data may be sent directly to the surgical stapler, which may identify the optimal location for tissue incision with the stapling device based on the tissue characteristics and / or disease state of the tissue or area with minimal vascular avoidance. For example, based on the area incised, the device may process and communicate to the surgical stapler which cartridge to use, e.g., 45mm or 60mm, and / or which color to use based on the thickness of the tissue collected in the jaws.

[0348] There are various types of surgical instruments used in surgical procedures. An "intelligent" or "smart" surgical instrument includes control circuitry (including a microcontroller having a processor and memory) and may include sensors (e.g., positioned on the end effector) that measure various derived parameters during the surgical task of the surgical procedure.

[0349] The hospital or operating room may be equipped with a centralized computing system to monitor or record various derived parameters from the surgical instruments used during the surgical procedure and, optionally, process those parameters to enhance control of the surgical instruments.

[0350] A potential drawback of a centralized computing system is that it requires sufficient memory and processing power to record or monitor large amounts of sensed data. Furthermore, if the centralized computing system fails (for any reason), data collection cannot occur during that time.

[0351] It is therefore an object of the present invention to provide a surgical system that alleviates these drawbacks.

[0352] The following is a non-exhaustive list of embodiments that may or may not be claimed. 1. A first surgical device configured for peer monitoring, comprising: a processor, the processor comprising: determining that a first surgical device is of a first surgical device type and that the first surgical device has the capability of being a monitoring surgical device in a surgical procedure, wherein determining that the first surgical device is of the first surgical device type and has the capability of being a monitoring surgical device for a second surgical device includes determining that the first surgical device is capable of performing monitoring and recording of surgical tasks being performed at the second surgical device; determining, based on a second surgical device type associated with the second surgical device, that the second surgical device has the capability to be monitored by the first surgical device; establishing a peer-to-peer connection with a second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device; and a first surgical device configured to use the established peer-to-peer connection with the second surgical device to monitor or record surgical information associated with the performance of a surgical task by the second surgical device. 2. The processor is configured to determine that the second surgical device has the capability to be monitored by the first surgical device; sending a discovery request indication to a second surgical device, the second surgical device being used in performing a surgical task associated with the surgical procedure; The first surgical device of embodiment 1, further comprising: the processor being configured to receive, from the second surgical device, an indication of a response message, the indication of the response message including information of the type and capabilities of the second surgical device associated with the second surgical device. 3. The first surgical device of embodiment 1, wherein the first surgical device type is a monitoring surgical device and the second surgical device type is a monitored surgical device type. 4. The first surgical device of embodiment 1, wherein the ability to be a monitoring surgical device or to be monitored includes having the ability to establish a peer-to-peer connection. 5. The first surgical device of embodiment 1, wherein the ability of the first surgical device to be a monitoring surgical device includes the monitoring surgical device having the ability to remotely access surgical information at a second surgical device. 6. The first surgical device of embodiment 5, wherein the ability of the first surgical device to be a monitoring surgical device includes the monitoring surgical device having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device. 7. The first surgical device of embodiment 1, wherein the first surgical device has the capability to be a monitoring surgical device and the second surgical device has the capability to be monitored by the first surgical device is determined based on negotiation between the first surgical device and the second surgical device. 8. The first surgical device of embodiment 1, wherein the second surgical device is a smart surgical hub. 9. The processor: establishing a second peer-to-peer connection with a third surgical device; The first surgical device of embodiment 1, further configured to initiate monitoring and recording of surgical tasks being performed at the second surgical device using the established peer-to-peer connection with the second surgical device, and at a third surgical device using the established second peer-to-peer connection. 10. The first surgical device of embodiment 9, wherein monitoring and recording a second surgical device using an established peer-to-peer connection with the second surgical device and monitoring and recording a third surgical device using the established second peer-to-peer connection are performed simultaneously. 11. A method of surgical information monitoring implemented on a first surgical device, the method comprising: determining that a first surgical device is of a first surgical device type and that the first surgical device has the capability of being a monitoring surgical device in a surgical procedure, wherein determining that the first surgical device is of the first surgical device type and has the capability of being a monitoring surgical device for a second surgical device includes determining that the first surgical device is capable of performing monitoring and recording of surgical tasks being performed at the second surgical device; determining, based on a second surgical device type associated with the second surgical device, that the second surgical device has the capability to be monitored by the first surgical device; establishing a peer-to-peer connection with a second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device; and using the established peer-to-peer connection with the second surgical device to monitor or record surgical information associated with the performance of a surgical task by the second surgical device. 12. transmitting a discovery request indication to a second surgical device, the second surgical device transmitting the discovery request indication for use in performing a surgical task associated with the surgical procedure; The method of embodiment 11, further comprising receiving an indication of a response message from a second surgical device, the indication of the response message including information of the type and capabilities of the second surgical device associated with the second surgical device. 13. The method of embodiment 11, wherein the first surgical device type is a monitoring surgical device and the second surgical device type is a monitored surgical device type. 14. The method of embodiment 11, wherein the ability to be a monitoring surgical device or to be monitored includes having the ability to establish a peer-to-peer connection. 15. The method of embodiment 11, wherein the ability of the first surgical device to be a monitoring surgical device includes the monitoring surgical device having the ability to remotely access surgical information at a second surgical device. 16. The method of embodiment 15, wherein the ability of the first surgical device to be a monitoring surgical device includes the monitoring surgical device having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device. 17. The method of claim 11, wherein the first surgical device has the capability to be a monitoring surgical device and the second surgical device has the capability to be monitored by the first surgical device is determined based on negotiation between the first surgical device and the second surgical device. 18. The method of embodiment 11, wherein the second surgical device is a smart surgical hub. 19. establishing a second peer-to-peer connection with a third surgical device; 12. The method of embodiment 11, further comprising monitoring and recording surgical tasks being performed by the second surgical device using the established peer-to-peer connection with the second surgical device, and by the third surgical device using the established second peer-to-peer connection. 20. The method of embodiment 19, wherein monitoring and recording a second surgical device using an established peer-to-peer connection with the second surgical device and monitoring and recording a third surgical device using the established second peer-to-peer connection are performed simultaneously.

[0353] [Embodiment] (1) a first surgical device configured for peer monitoring, the first surgical device comprising: a processor, the processor comprising: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device; and a first surgical device configured to monitor or record surgical information associated with the performance of the surgical task by the second surgical device using the established peer-to-peer connection. (2) The processor: 2. The first surgical device of embodiment 1, further configured to obtain the information regarding the second surgical device directly from the second surgical device. (3) The processor: processing the information associated with the second surgical device to generate proposed changes to control settings for the second surgical device; adjusting a control setting of the second surgical device; or 3. The first surgical device of embodiment 1 or 2, further configured to send recommendations for changing the control settings to the second surgical device. (4) The processor: processing the information associated with the second surgical device to generate proposed changes to control settings for the first surgical device; A first surgical device described in any of embodiments 1 to 3, further configured to adjust control settings of the first surgical device based on the proposed changes. (5) the processor being configured to determine that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; sending a discovery request indication to the second surgical device; receiving an indication of a response message from the second surgical device; or receiving a discovery request indication from the second surgical device; A first surgical device according to any one of embodiments 1 to 4, further comprising the processor being configured to send an instruction in a response message to the second surgical device.

[0354] (6) A first surgical device as described in embodiment 5, wherein the instructions in the response message may include information on the type and capabilities of the surgical device. (7) A first surgical device described in any of embodiments 1 to 6, wherein determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device includes determining that one of the first surgical device and the second surgical device is capable of accessing or receiving data from the other. (8) A first surgical device as described in embodiment 7, wherein the determination of which of the first surgical device and the second surgical device will be the monitoring device and which will be the monitored device is based on the capabilities of the first surgical device and the second surgical device. (9) A first surgical device as described in embodiment 7 or 8, wherein the ability of the first surgical device to be a monitoring surgical device includes having the ability to remotely set parameters of the second surgical device based on accessed surgical information in the second surgical device. (10) A first surgical device described in any of embodiments 7 to 9, wherein the first surgical device has the capability to be the monitoring surgical device and the second surgical device has the capability to be monitored by the first surgical device is determined based on negotiation between the first surgical device and the second surgical device.

[0355] (11) The first surgical device according to any one of embodiments 1 to 10, wherein the second surgical device is a surgical hub. (12) The processor: establishing a second peer-to-peer connection with a third surgical device; A first surgical device described in any of embodiments 1 to 11, further configured to initiate monitoring and recording of the second surgical device using the established peer-to-peer connection with the second surgical device, and to initiate monitoring and recording of the third surgical device using the established second peer-to-peer connection. (13) A first surgical device as described in embodiment 12, wherein monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device and monitoring and recording the third surgical device using the established second peer-to-peer connection are performed simultaneously. (14) A first surgical device configured to be peer monitored, the first surgical device comprising: a processor, the processor comprising: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the second surgical device to monitor and record surgical tasks performed by the first surgical device; performing a surgical task; and transmitting surgical information associated with the performance of the surgical task to the second surgical device using the established peer-to-peer connection. (15) A system configured for peer monitoring, the system comprising: a hub or edge device having a processor, the processor comprising: establishing a connection with at least a first surgical device and a second surgical device; assigning a monitoring role to the first surgical device based on one or more of the types of the first and second surgical devices, the tasks performed in a surgical procedure, and the capabilities of the first and second surgical devices; configured to assign a monitored role to the second surgical device based on at least one of the types of the first and second surgical devices, the tasks performed in a surgical procedure, and the capabilities of the first and second surgical devices; the first surgical device is configured to establish a peer-to-peer connection with a second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device; and The system, wherein the first surgical device is configured to monitor or record surgical information associated with the performance of the surgical task by the second surgical device using the established peer-to-peer connection.

[0356] (16) A method of peer-to-peer surgical information monitoring implemented on a first surgical device, the method comprising: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks on the second surgical device; and monitoring or recording surgical information associated with the second surgical device using the established peer-to-peer connection with the second surgical device. (17) The method of embodiment 16, wherein the information regarding the second surgical device is obtained directly from the second surgical device. (18) processing the information associated with the second surgical device to generate suggested changes to control settings for the second surgical device; adjusting a control setting of the second surgical device; or 18. The method of embodiment 16 or 17, further comprising sending a recommendation for changing the control settings to the second surgical device. (19) processing the information associated with the second surgical device to generate suggested changes to control settings for the first surgical device; A method according to any one of embodiments 16 to 18, further comprising adjusting control settings of the first surgical device based on the proposed changes. (20) determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; sending a discovery request indication to the second surgical device; and receiving an indication of a reply message from the second surgical device; or receiving a discovery request indication from the second surgical device; and A method according to any one of embodiments 16 to 19, further comprising sending a response message instruction to the second surgical device.

[0357] (21) The method of embodiment 20, wherein the instructions in the response message may include information on the type and capabilities of the surgical device. (22) A method according to any of embodiments 16 to 21, wherein determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device includes determining that one of the first surgical device and the second surgical device is capable of accessing or receiving data from the other. (23) The method of embodiment 22, wherein the determination of which of the first surgical device and the second surgical device will be the monitoring device and which will be the monitored device is based on the capabilities of the first surgical device and the second surgical device. (24) The method of embodiment 22 or 23, wherein the ability of the first surgical device to be a monitoring surgical device includes having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device. (25) A method according to any of embodiments 22 to 24, wherein the first surgical device has the capability to be the monitoring surgical device and the second surgical device has the capability to be monitored by the first surgical device is determined based on negotiation between the first surgical device and the second surgical device.

[0358] (26) The method of any one of embodiments 16 to 25, wherein the second surgical device is a surgical hub. (27) establishing a second peer-to-peer connection with a third surgical device; A method according to any one of embodiments 16 to 26, further comprising monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device, and monitoring and recording the third surgical device using the established second peer-to-peer connection. (28) The method of embodiment 27, wherein monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device and monitoring and recording the third surgical device using the established second peer-to-peer connection are performed simultaneously. (29) A computing program that, when executed by a processor on a first surgical device, causes the processor to perform the method described in any one of embodiments 16 to 28.

Claims

1. A first surgical device configured for peer monitoring, the first surgical device comprising: a processor, the processor comprising: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device; and a first surgical device configured to use the established peer-to-peer connection to monitor or record surgical information associated with the performance of the surgical task by the second surgical device.

2. the processor: The first surgical device of claim 1 , further configured to obtain the information regarding the second surgical device directly from the second surgical device.

3. the processor: processing the information associated with the second surgical device to generate proposed changes to control settings for the second surgical device; adjusting a control setting of the second surgical device; or The first surgical device of claim 1 or 2, further configured to transmit recommendations for changing the control settings to the second surgical device.

4. the processor: processing the information associated with the second surgical device to generate proposed changes to control settings for the first surgical device; The first surgical device of claim 1 , further configured to adjust control settings of the first surgical device based on the proposed changes.

5. The processor being configured to determine that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device, sending a discovery request indication to the second surgical device; receiving an indication of a response message from the second surgical device; or receiving a discovery request indication from the second surgical device; The first surgical device of claim 1 , further comprising the processor configured to send an indication of a reply message to the second surgical device.

6. The first surgical device of claim 5 , wherein the indication of the response message can include surgical device type and capability information.

7. 10. The first surgical device of claim 1, wherein determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device comprises determining that one of the first surgical device and the second surgical device is capable of accessing or receiving data from the other.

8. 8. The first surgical device of claim 7, wherein a determination of which of the first surgical device and the second surgical device will be the monitoring device and which will be the monitored device is based on the capabilities of the first surgical device and the second surgical device.

9. 9. The first surgical device of claim 7 or 8, wherein the ability of the first surgical device to be a monitoring surgical device includes having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device.

10. 8. The first surgical device according to claim 7, wherein the first surgical device has the capability to be the monitoring surgical device and the second surgical device has the capability to be monitored by the first surgical device is determined based on negotiation between the first surgical device and the second surgical device.

11. The first surgical device of claim 1 , wherein the second surgical device is a surgical hub.

12. the processor: establishing a second peer-to-peer connection with a third surgical device; 10. The first surgical device of claim 1, further configured to initiate monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device, and to initiate monitoring and recording the third surgical device using the established second peer-to-peer connection.

13. 13. The first surgical device of claim 12, wherein monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device and monitoring and recording the third surgical device using the established second peer-to-peer connection are performed simultaneously.

14. A first surgical device configured to be peer monitored, the first surgical device comprising: a processor, the processor comprising: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the second surgical device to monitor and record surgical tasks performed by the first surgical device; performing a surgical task; and transmitting surgical information associated with the performance of the surgical task to the second surgical device using the established peer-to-peer connection.

15. 1. A system configured for peer monitoring, the system comprising: a hub or edge device having a processor, the processor comprising: establishing a connection with at least a first surgical device and a second surgical device; assigning a monitoring role to the first surgical device based on one or more of the types of the first and second surgical devices, the tasks performed in a surgical procedure, and the capabilities of the first and second surgical devices; configured to assign a monitored role to the second surgical device based on at least one of the types of the first and second surgical devices, the tasks performed in a surgical procedure, and the capabilities of the first and second surgical devices; the first surgical device is configured to establish a peer-to-peer connection with a second surgical device for the first surgical device to monitor and record surgical tasks performed by the second surgical device; and The system, wherein the first surgical device is configured to monitor or record surgical information associated with the performance of the surgical task by the second surgical device using the established peer-to-peer connection.

16. 1. A method of peer-to-peer surgical information monitoring implemented on a first surgical device, the method comprising: determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device; establishing a peer-to-peer connection with the second surgical device, the peer-to-peer connection being established between the first surgical device and the second surgical device for the first surgical device to monitor and record surgical tasks on the second surgical device; and monitoring or recording surgical information associated with the second surgical device using the established peer-to-peer connection with the second surgical device.

17. The method of claim 16 , wherein the information regarding the second surgical device is obtained directly from the second surgical device.

18. processing the information associated with the second surgical device to generate proposed changes to control settings for the second surgical device; adjusting a control setting of the second surgical device; or 18. The method of claim 16 or 17, further comprising transmitting the recommendation for changing the control settings to the second surgical device.

19. processing the information associated with the second surgical device to generate proposed changes to control settings for the first surgical device; The method of claim 16, further comprising adjusting a control setting of the first surgical device based on the proposed change.

20. Determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device includes: sending a discovery request indication to the second surgical device; and receiving an indication of a reply message from the second surgical device; or receiving a discovery request indication from the second surgical device; and The method of claim 16, further comprising sending an indication of a reply message to the second surgical device.

21. The method of claim 20 , wherein the indications in the response message may include surgical device type and capability information.

22. 17. The method of claim 16, wherein determining that the first surgical device is capable of establishing a peer-to-peer connection with a second surgical device comprises determining that one of the first surgical device and the second surgical device is capable of accessing or receiving data from the other.

23. 23. The method of claim 22, wherein determining which of the first surgical device and the second surgical device will be the monitoring device and which will be the monitored device is based on the capabilities of the first surgical device and the second surgical device.

24. 24. The method of claim 22 or 23, wherein the ability of the first surgical device to be a monitoring surgical device includes having the ability to remotely set parameters of the second surgical device based on accessed surgical information at the second surgical device.

25. 23. The method of claim 22, wherein the first surgical device has the capability to be the monitoring surgical device and the second surgical device has the capability to be monitored by the first surgical device is determined based on negotiation between the first surgical device and the second surgical device.

26. The method of claim 16 , wherein the second surgical device is a surgical hub.

27. establishing a second peer-to-peer connection with a third surgical device; 17. The method of claim 16, further comprising: monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device; and monitoring and recording the third surgical device using the established second peer-to-peer connection.

28. 28. The method of claim 27, wherein monitoring and recording the second surgical device using the established peer-to-peer connection with the second surgical device and monitoring and recording the third surgical device using the established second peer-to-peer connection are performed simultaneously.

29. A computing program that, when executed by a processor on a first surgical device, causes the processor to perform the method of claim 16.

Images