Contextual awareness of instrument location and user personalization for controlling displays

Abstract

A surgical hub may be provided for controlling a display using situational awareness of the medical instrument. The location of the user, the medical instrument, and the operating room may be determined. Contextual data associated with the medical instrument may be determined based on the location of the user, the medical instrument, and the operating room. Display commands may be sent to the display that may instruct the display to be configured according to the contextual data associated with the medical instrument.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is related to the following, the contents of each of which are incorporated herein by reference: United States patent application entitled "METHOD FOR OPERATING TIERED OPERATION MODES IN A SURGICAL SYSTEM," filed herewith, having attorney docket number END9287USNP1; United States patent application entitled "SHARED SITUATIONAL AWARENESS OF THE DEVICE ACTUATOR ACTIVITY TO PRIORITIZE CERTAIN ASPECTS OF DISPLAYED INFORMATION," filed herewith, having attorney docket number END9288USNP2; United States patent application entitled "MONITORING OF USER VISUAL GAZE TO CONTROL WHICH DISPLAY SYSTEM DISPLAYS THE PRIMARY INFORMATION," filed herewith, having attorney docket number END9288USNP3; United States patent application entitled "RECONFIGURATION OF DISPLAY SHARING," filed herewith, having attorney docket number END9288USNP4; and United States patent application entitled "CONTROL A DISPLAY OUTSIDE THE STERILE FIELD FROM A DEVICE WITHIN THE STERILE FIELD," filed herewith, having attorney docket number END9288USNP5. [Background technology]

[0002] Surgical systems often incorporate imaging systems that can enable clinicians to view the surgical site and / or one or more portions thereof on one or more displays, such as, for example, monitors. The displays may be local to the surgical theater and / or remote. The imaging system may include a scope with a camera that views the surgical site and transmits the view to a display viewable by the clinician. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngological-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. Imaging systems may be limited by the information they can recognize and / or convey to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery with a particular imaging system. Additionally, certain imaging systems may not be able to communicate and / or convey certain information to the clinician during surgery. Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment of the present invention, a surgical hub and / or medical instrument for controlling a display using situational awareness is provided. The surgical hub and / or medical instrument may include a memory and a processor. The processor may be configured to perform several actions. The location of a user, a medical instrument, and an operating room may be determined. Contextual data (e.g., context information) associated with the medical instrument may be determined based on the location of the user, the medical instrument, and the operating room. Display instructions may be sent to the display, which may instruct the display to be configured according to the contextual data (e.g., context information) associated with the medical instrument. The display may be a primary display or a secondary display.

[0004] As examples above and below, the surgical hub can control the display configuration based on the situational awareness of the instrument user, the instrument itself, and the location in the operating room. Data can be collected from cameras and sensors to provide the hub with situational awareness of the operating room. As a result, the surgical hub can turn off the display after a user places an instrument in a given location, delete or wipe instrument data, determine that cleaning is necessary and display this, detect another instrument, determine that an instrument in a surgical procedure is about to be switched and adjust the display data accordingly, and determine that an instrument is about to be used to perform a surgical task and display data to assist with this. As a result, the efficiency and safety of the surgical procedure are improved, fewer display commands are required from surgical team members, saving personnel time, and reducing the number of distractions and opportunities for error. Additionally, the ability to adapt the display configuration and content to the location, user, and instrument increases the adaptability of the surgical hub, allowing it to be used in a wider variety of procedures. Specific instructions (e.g., cleaning instructions, reloading instructions, surgical procedure instructions, or instrumentation instructions) can be displayed to the user at the precise time and adjusted in real time, further improving efficiency by saving time and safety by reducing errors. The display can be updated during the surgical procedure based on how the instrument is being manipulated using parameters such as device orientation and flipping so that data can still be displayed in the correct orientation and more quickly understood by the surgical team. Overall, the surgical hub improves the efficiency and safety of surgical procedures, as well as greater adaptability to new procedures and methods.

[0005] According to further embodiments of the present invention, the following examples are provided. 1. A surgical hub for controlling a display, the surgical hub comprising: a processor, determining the location of a user, medical equipment, and an operating room; determining contextual data associated with the medical instrument based on the user, the medical instrument, and a location within the operating room; and transmitting display instructions to the display instructing the display to be configured according to contextual data associated with the medical instrument.

[0006] For example, in Example 1, the surgical hub is configured to use situational awareness of the medical instrument to control the display.

[0007] For example, in Example 1, the surgical hub may be configured to connect to a medical instrument and a display.

[0008] For example, in Example 1, the medical instrument may be at least one of a laparoscope, an endoscope, a thoracoscope, a laparoscopic surgical instrument, an electrosurgical instrument, an ultrasonic surgical instrument, and / or a surgical stapling instrument.

[0009] For example, in Example 1, the context data associated with the medical device may include at least one of an indication that a user is controlling the medical device, an indication that the medical device is at a particular location, an indication that the user is at a particular location, an indication that the user is more than a threshold distance away from the location, an indication that the user is more than a threshold distance away from the device, an indication of the location type or purpose of the location, an indication that the user is replacing a medical device such as a medical instrument, an indication that the location is proximate to or inside a patient, an indication that the location is a surgical site and / or an indication that the location is inside a trocar, user feedback, parameters for the medical device adjusted by the user, a wait time, a force-to-fire (FTF) parameter, a clamp compression parameter, an indication that a cartridge is loaded, a cartridge status, an indication of an enabled control, an indication of a disabled medical device control, a battery power level, and a status of the medical device.

[0010] For example, in Example 1, the locations may include at least one of the surgical site, a back table, an instrument storage table or area, a location within a sterile field, a location within a non-sterile field, a storage area, a preparation area, an area away from the patient, a surgical table, and a cleaning station.

[0011] 2. A surgical hub as described in Example 1, wherein the processor is further configured to determine display content related to contextual data associated with the medical instrument.

[0012] 3. A surgical hub as described in Example 1 or 2, wherein the display instructions further include display content.

[0013] For example, in Examples 1-3, transmitting a display instruction to the display instructing the display to be configured in accordance with context data associated with the medical device may include controlling display of the display content or the display content based on the context information, transmitting the display content or the display content to the display, and transmitting a display instruction to the display instructing the display to display the display content or the display content in accordance with the context data associated with the medical device.

[0014] For example, in Examples 1-3, sending display instructions to the display instructing the display to display content or display content according to context data associated with the medical device may include indicating that the medical device should be powered off, indicating that it should be powered on, deleting device, patient, and / or surgical data, indicating that the medical device should be in a cleaning mode, providing cleaning instructions for the medical device to the user, indicating that a second medical device has been replaced with the medical device, adding first device data associated with the medical device, and adding second device data associated with the second medical device. 3. The method may include transmitting display instructions to the display instructing the display to at least one of: deleting, showing instrument data, showing instrument data from the medical instrument, showing instructions to use the medical instrument, showing surgical instructions or instructions to use the medical instrument, indicating that a user is controlling the medical instrument and that the medical instrument is being used to perform a surgical task, showing instrument data based on one or more of an orientation of the medical instrument, indicating a handedness of the user, indicating a level of reversal of the medical instrument, showing a reload instruction for the medical instrument, and showing a reconfiguration instruction for the surgical instrument. determining that the medical device is in position; determining that the user is more than a threshold distance away from the location; determining that the location indicates that the medical device is to be powered off; A surgical hub as described in any one of Examples 1 to 3, configured to determine contextual data associated with the medical instrument based on the user, the medical instrument, and its location within the operating room by setting a display instruction to indicate that the medical instrument should be powered off.

[0015] For example, in Examples 1-4, sending a display instruction to the display instructing the display to configure according to the context data associated with the medical device may include setting the display instruction to indicate that the medical device should be powered off, or sending a display instruction to the display indicating that the medical device should be powered off or instructing the display to power off.

[0016] For example, in Example 4, determining that the location indicates that the medical device is to be powered off may include determining that the medical device is to be powered off as a result of the user being more than a threshold distance away from the location.

[0017] For example, in any of Examples 1 to 4, the processor may be configured to: determine that the user is more than a threshold distance away from the medical device; and determine that the medical device will be powered off as a result of the user being more than the threshold distance away from the medical device; determine context data associated with the medical device based on the user, the medical device, and their location within the operating room; and send display instructions to the display instructing the display to configure the display in accordance with the context data associated with the medical device by setting a display instruction to indicate that the medical device should be powered off or sending a display instruction to the display indicating that the medical device should be powered off or instructing the display to power off.

[0018] 5. A surgical hub as described in any one of Examples 1 to 4, wherein a display command to the display instructing the display to be configured according to context data associated with a medical instrument causes the display to turn off or delete the instrument data.

[0019] For example, in any of Examples 1-4, sending a display instruction to the display instructing the display to configure according to context data associated with the medical device may include sending a display instruction to the display causing the display to turn off or delete the device data.

[0020] 6. The processor: determining that the medical device and the user are within a threshold distance of a location; determining that the position indicates that the medical instrument is to be cleaned; A surgical hub as described in any one of Examples 1 to 5, configured to determine contextual data associated with the medical instrument based on the user, the medical instrument, and its location within the operating room by setting a display instruction to indicate that the medical instrument should be in cleaning mode.

[0021] For example, in Examples 1-6, sending display instructions to the display instructing the display to configure according to context data associated with the medical device may include setting the display instructions to indicate that the medical device should be in a cleaning mode, and sending display instructions to the display instructing the display to indicate that the medical device should be in a cleaning mode.

[0022] 7. A surgical hub described in any one of Examples 1 to 6, wherein display instructions to the display instructing the display to be configured according to context data associated with the medical instrument cause the display to provide cleaning instructions to the user for the medical instrument.

[0023] For example, in any of Examples 1-7, sending display instructions to the display instructing the display to configure according to context data associated with the medical device may include sending display instructions to the display instructing or causing the display to provide cleaning instructions for the medical device to a user.

[0024] 8. The medical device is a first medical device, and the processor: determining that the first medical device, the second medical device, and the user are within a threshold distance of a location; determining that a user is replacing a second medical device with a first medical device; and setting a display instruction to indicate that a second medical instrument has been replaced with the first medical instrument. A surgical hub as described in any one of Examples 1 to 7, configured to determine contextual data for the first medical instrument based on the user, the first medical instrument, and a location within the operating room.

[0025] For example, in any of Examples 1-8, sending display instructions to the display instructing the display to configure according to the context data associated with the medical device may include setting the display instructions to indicate that a second medical device is being replaced with the first medical device, or sending display instructions to the display instructing the display to indicate that the second medical device is being replaced with the first medical device.

[0026] For example, in any of Examples 1-8, the surgical hub may be configured to connect to a first medical instrument, a second medical instrument, and a display.

[0027] 9. A surgical hub as described in any one of Examples 1 to 8, wherein display instructions to the display instructing the display to be configured according to context data associated with a first medical instrument cause the display to add first instrument data associated with the first medical instrument and remove second instrument data associated with a second medical instrument.

[0028] For example, in Examples 1-9, sending display instructions to the display instructing the display to configure according to context data associated with the medical device may include instructing the display to add first device data associated with a first medical device and remove second device data associated with a second medical device.

[0029] 10. The context data is first context data, the display instructions are first display instructions, the display is a first display, and the processor: determining second context data associated with the second medical instrument based on the user, the second medical instrument, and a location within the operating room; A surgical hub as described in any one of Examples 1 to 9, further configured to: turn off the second display or send a second display instruction to the second display instructing the second display to configure itself according to the second context data by displaying one or more of instructions to reload the second medical instrument, instructions to clean the second medical instrument, or surgical instructions for using the second medical instrument.

[0030] For example, in Example 10, sending the second display instructions to the second display may include powering off, indicating that the second medical device should be powered off, displaying one or more of instructions to reload the second medical device, instructions to clean the second medical device, or instructions to perform a surgical procedure using the second medical device, deleting the device, patient, and / or surgical data, indicating that the medical device should be in a cleaning mode, indicating that the second medical device has been replaced with a medical device, adding the first device data associated with the medical device, and displaying the second device data to the second medical device. The method may include sending second display instructions to the second display instructing the second display to delete associated second instrument data, show the instrument data, show instrument data from the medical instrument, show instructions to use the medical instrument, show that a user is controlling the medical instrument and that the medical instrument is being used to perform a task of a surgical procedure, show instrument data based on one or more of the orientation of the medical instrument, show a handedness of the user, show a level of reversal of the medical instrument, and show reconfiguration instructions for the surgical instrument.

[0031] 11. A surgical hub described in any one of Examples 1 to 9, wherein a display instruction to the display instructing the display to be configured according to context data of the medical instrument causes the display to show instrument data or instructions for using the medical instrument.

[0032] For example, in Examples 1-9, sending display instructions to the display instructing the display to be configured according to context data associated with the medical device may include instructing the display to show medical device data and / or device data from the medical device and / or instructions for using the medical device.

[0033] 12. The user is a first user, and the processor: determining that the medical device is being moved from the second user to the first user within a threshold distance of the location; determining that the location is near the patient; A surgical hub as described in any one of Examples 1 to 11, configured to determine contextual data for the medical instrument based on the first user, the medical instrument, and its location within the operating room by setting display instructions to indicate that a first user is controlling the medical instrument and that the medical instrument is used to perform a surgical procedure task.

[0034] For example, in Examples 1-12, sending display instructions to the display instructing the display to configure according to the context data associated with the medical instrument may include setting the display instructions to indicate that a first user is controlling the medical instrument and that the medical instrument is used to perform a surgical task, or sending display instructions to the display instructing the display to indicate that a first user is controlling the medical instrument and that the medical instrument is used to perform a surgical task. Determining that the location is near the patient may include determining that the location is in close proximity to the patient, such as within one meter of the patient, determining that the location is inside the patient, and determining that the location is inside a trocar.

[0035] For example, in Example 12, the second user may be a nurse, a delivery nurse, a medical technician, a surgeon's assistant, and / or a medical professional. The first user may be a doctor and / or a surgeon.

[0036] 13. A surgical hub described in any one of Examples 1 to 12, wherein the context data indicates that a user is controlling a medical instrument and the display instructions include instructions to cause the display to show one or more of instrument data, medical instrument instructions, and surgical procedure instructions.

[0037] For example, in any of Examples 1-13, sending display instructions to the display instructing the display to configure according to context data associated with the medical device may include sending display instructions to the display causing or instructing the display to display one or more of device data, such as first and / or second medical device data, medical device instructions, such as first and / or second medical device instructions, and / or surgical procedure instructions.

[0038] 14. A surgical hub described in any one of Examples 1 to 13, wherein the context data indicates that a user is controlling a medical instrument, and the display instructions include instructions to cause the display to show instrument data based on one or more of the orientation of the medical instrument, the user's handedness, and the level of inversion of the medical instrument.

[0039] For example, in any of Examples 1-14, sending display instructions to the display instructing the display to configure according to context data associated with the medical device may include sending display instructions to the display instructing or causing the display to show device data based on one or more of the orientation of the medical device, the user's handedness, and the level of inversion of the medical device.

[0040] 15. A surgical hub described in any one of Examples 1 to 14, wherein the processor is configured to determine the positions of the user, medical instruments, and within the operating room using one or more of a camera, sensors in the operating room, sensors associated with the user, sensors associated with the medical instruments, and a wearable device.

[0041] For example, in any of Examples 1-15, the surgical hub may include a situational awareness system configured to derive a user, a medical instrument, and a location from situational awareness data. The situational awareness system may include or be configured to connect to at least one hardware for transmitting the situational awareness data to the situational awareness system. The hardware may include at least one of a camera, a sensor in the operating room, a sensor associated with the user, a sensor associated with the medical instrument, and / or a wearable device.

[0042] 16. A surgical hub described in any one of Examples 1 to 15, wherein the user is one or more of a patient, a healthcare provider, a doctor, a nurse, a delivery nurse, and a medical technician.

[0043] For example, in any of Examples 1-16, the user may be one or more of a healthcare provider, a doctor, a nurse, a surgical nurse, a medical technician, a surgeon, a surgical assistant, and / or a medical professional.

[0044] 17. The surgical hub of any one of Examples 1-16, wherein the display is a primary display or a secondary display.

[0045] 18. A method for a surgical hub to control a display, the method comprising: determining the location of a user, medical equipment, and an operating room; determining contextual data associated with the medical instrument based on the user, the medical instrument, and a location within the operating room; and transmitting display instructions to the display instructing the display to be configured according to contextual data associated with the medical device.

[0046] All comments above regarding Example 1 apply mutatis mutandis to Example 18. For example, in Example 18, the method may include using situational awareness of the medical instrument to control a display.

[0047] 19. The method of example 18, further comprising determining display content related to contextual data associated with the medical device.

[0048] All comments above regarding Example 2 apply mutatis mutandis to Example 19 20. The method of any one of examples 18 or 19, wherein the display instructions further include display content.

[0049] All comments above regarding Example 3 apply mutatis mutandis to Example 20. [Brief explanation of the drawings]

[0050] [Figure 1] FIG. 1 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 2] 1 is a surgical system used to perform a surgical procedure in an operating room, according to at least one aspect of the present disclosure. [Figure 3] 1 is a surgical hub paired with a visualization system, a robotic system, and an intelligent instrument, according to at least one aspect of the present disclosure. [Figure 4] 1 illustrates a surgical data network comprising a modular communications hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud, in accordance with at least one embodiment of the present disclosure. [Figure 5] 1 illustrates a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 6] 1 illustrates a surgical hub comprising multiple modules coupled to a modular control tower, according to at least one embodiment of the present disclosure. [Figure 7] 1 illustrates a logic diagram of a control system for a surgical instrument or tool, according to at least one aspect of the present disclosure. [Figure 8]1 illustrates a surgical instrument or tool with multiple motors that can be activated to perform various functions, according to at least one aspect of the present disclosure. [Figure 9] FIG. 1 is a diagram of a context-aware surgical system according to at least one aspect of the present disclosure. [Figure 10] 1 illustrates an exemplary timeline of a surgical procedure and inferences that a surgical hub can make from data detected at each step in the surgical procedure, in accordance with at least one aspect of the present disclosure. [Figure 11] FIG. 1 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 12] FIG. 1 is a block diagram illustrating a functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 13] 1 shows a block diagram of a computer-implemented interactive surgical system configured to adaptively generate control program updates for modular devices, in accordance with at least one aspect of the present disclosure. [Figure 14] 1 illustrates a surgical system including a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter, according to at least one aspect of the present disclosure. [Figure 15A] 10 illustrates an exemplary flow for determining an operating mode and operating in the determined mode. [Figure 15B] 10 illustrates an exemplary flow for changing the operating mode. [Figure 16] 1 shows the primary display of the surgical hub. [Figure 17] 1 shows an example of a primary display for a surgical hub. [Figure 18] Shown are four wide-angle view images of the surgical site at four separate times during the procedure. [Figure 19] 1 shows an example of an augmented video image of a pre-operative video image augmented with data identifying the displayed element. [Figure 20]1 shows an example flow diagram of a process for displaying one or more images. [Figure 21] 1 shows a schematic diagram of a beam source and associated beam detector system utilized as a device control mechanism in an operating room, in accordance with at least one aspect of the present disclosure. [Figure 22A] 1 illustrates various types of sterile field control and data entry consoles in accordance with at least one aspect of the present disclosure. [Figure 22B] 1 illustrates various types of sterile field control and data entry consoles in accordance with at least one aspect of the present disclosure. [Figure 22C] 1 illustrates various types of sterile field control and data entry consoles in accordance with at least one aspect of the present disclosure. [Figure 22D] 1 illustrates various types of sterile field control and data entry consoles in accordance with at least one aspect of the present disclosure. [Figure 22E] 1 illustrates various types of sterile field control and data entry consoles in accordance with at least one aspect of the present disclosure. [Figure 22A] A single-zone sterile field control and data entry console is shown. [Figure 22B] A multi-zone sterile field control and data entry console is shown. [Figure 22C] A tethered sterile field control and data entry console is shown. [Figure 22D] A battery-powered sterile field control and data entry console is shown. [Figure 22E] A battery-powered sterile field control and data entry console is shown. [Figure 23A] 1 illustrates a sterile field console in use in a sterile field during a surgical procedure, according to at least one embodiment of the present disclosure; 2 illustrates the sterile field console positioned in the sterile field near two surgeons engaged in the procedure; [Figure 23B] 1 illustrates a sterile field console in use in a sterile field during a surgical procedure, with one of the surgeons tapping on the sterile field console's touchscreen, according to at least one embodiment of the present disclosure. [Figure 24]1 illustrates a standard technique for estimating vessel path and depth and device trajectory, according to at least one aspect of the present disclosure. [Figure 25A] 1A-1C illustrate multiple real-time views of images of virtual anatomical details for dissection, in accordance with at least one embodiment of the present disclosure. [Figure 25B] 1A-1C illustrate multiple real-time views of images of virtual anatomical details for dissection, in accordance with at least one embodiment of the present disclosure. [Figure 25C] 1A-1C illustrate multiple real-time views of images of virtual anatomical details for dissection, in accordance with at least one embodiment of the present disclosure. [Figure 25D] 1A-1C illustrate multiple real-time views of images of virtual anatomical details for dissection, in accordance with at least one embodiment of the present disclosure. [Figure 26A] 10A-10C illustrate touchscreen displays that may be used within a sterile field according to aspects of the present disclosure, and 10B illustrate an image of a surgical site displayed on the touchscreen display in portrait mode. [Figure 26B] 1 illustrates a touchscreen display that may be used within a sterile field according to aspects of the present disclosure, shown rotated in landscape mode with the surgeon using his index finger to scroll through the images in the direction of the arrows. [Figure 26C] 10 illustrates a touchscreen display that may be used within a sterile field, according to aspects of the present disclosure, showing a surgeon pinching open the image in the direction of the arrows to zoom in. [Figure 26D] 10 illustrates a touchscreen display that may be used within a sterile field, according to aspects of the present disclosure, showing a surgeon pinching the image closed in the direction of the arrows to zoom out. [Figure 26E]1A-1C illustrate touchscreen displays that may be used within a sterile field according to embodiments of the present disclosure, and are shown rotated in two directions indicated by arrows so that the surgeon can view images in different orientations. [Figure 27] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for communicating from inside a sterile field to a device located outside the sterile field, in accordance with at least one aspect of the present disclosure. [Figure 28] 1 illustrates a second information layer overlaying a first information layer, according to at least one embodiment of the present disclosure. [Figure 29] FIG. 1 illustrates a perspective view of a surgeon using a surgical instrument including a handle assembly housing and a wireless circuit board during a surgical procedure, with the surgeon wearing safety glasses, in accordance with at least one embodiment of the present disclosure. [Figure 30] 1 illustrates a method for identifying surgical data associated with a fault event and communicating the identified surgical data to a cloud-based system on a priority basis, according to at least one aspect of the present disclosure. [Figure 31] 10 illustrates ultrasonic pinging of an operating room wall to determine the distance between the surgical hub and the operating room wall, in accordance with at least one aspect of the present disclosure. [Figure 32] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for surgical hub pairing with a surgical device of a surgical system located within the confines of an operating room, in accordance with at least one aspect of the present disclosure. [Figure 33] FIG. 10 is a process logic flow diagram illustrating a control program or logic configuration for selectively making and breaking connections between devices of a surgical system in accordance with at least one aspect of the present disclosure. [Figure 34] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for selectively re-evaluating operating room boundaries after detecting a new device, in accordance with at least one aspect of the present disclosure. [Figure 35]FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for selectively re-evaluating operating room boundaries after a paired device is disconnected, in accordance with at least one aspect of the present disclosure. [Figure 36] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for re-evaluating the boundaries of the operating room by the surgical hub after detecting a change in the position of the surgical hub, in accordance with at least one aspect of the present disclosure. [Figure 37] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for selectively forming connections between devices of a surgical system in accordance with at least one aspect of the present disclosure. [Figure 38] FIG. 10 is a process logic flow diagram illustrating a control program or logic configuration for selectively making and breaking connections between devices of a surgical system in accordance with at least one aspect of the present disclosure. [Figure 39] 1 illustrates a surgical hub pairing a first device and a second device of a surgical system in an operating room, according to at least one aspect of the present disclosure. [Figure 40] 1 illustrates a surgical hub unpairing a first device and a second device of a surgical system in an operating room and pairing the first device with a third device in the operating room, according to at least one aspect of the present disclosure. [Figure 41] FIG. 10 is a process logic flow diagram illustrating a control program or logic configuration for making and breaking connections between devices of a surgical system in an operating room during a surgical procedure based on the progression of steps in the surgical procedure, in accordance with at least one aspect of the present disclosure. [Figure 42] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for overlaying information from one or more still frames of a live stream of a remote surgical site onto the live stream, in accordance with at least one aspect of the present disclosure. [Figure 43] FIG. 1 is a process logic flow diagram illustrating a control program or logic configuration for distinguishing surgical steps of a surgical procedure, in accordance with at least one aspect of the present disclosure. [Figure 44]FIG. 1 is a process logic flow diagram illustrating a control program or logic configuration for distinguishing surgical steps of a surgical procedure, in accordance with at least one aspect of the present disclosure. [Figure 45] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for identifying a staple cartridge from information derived from one or more still frames of staples deployed from the staple cartridge into tissue in accordance with at least one embodiment of the present disclosure. [Figure 46] FIG. 1 is a partial view of a surgical system in an operating room, the surgical system including a surgical hub having an imaging module in communication with an imaging device at a remote surgical site, according to at least one embodiment of the present disclosure. [Figure 47] 1 illustrates a partial artificial timeline of a surgical procedure performed in an operating room via a surgical system, according to at least one aspect of the present disclosure. [Figure 48] 1 illustrates interaction between two surgical hubs in different operating rooms (“OR1” and “OR3”), according to at least one embodiment of the present disclosure. [Figure 49] 1 illustrates a secondary display in an operating room ("OR3") showing a surgical site for a colorectal procedure, according to at least one embodiment of the present disclosure. [Figure 50] 1 illustrates a personal interface or tablet within OR1 displaying the surgical site of OR3, according to at least one embodiment of the present disclosure. [Figure 51] 1 shows a close-up view of OR3's surgical site displayed on OR1's primary display, according to at least one embodiment of the present disclosure. [Figure 52] 1 illustrates a personal interface or tablet displaying a layout of OR1, showing available displays, according to at least one embodiment of the present disclosure. [Figure 53] 1 illustrates a recommendation of a transection location for a surgical site in OR3 made by a surgeon in OR1 via a personal interface or tablet in OR1, according to at least one embodiment of the present disclosure. [Figure 54A]FIG. 10 is a logic flow diagram of a process for controlling modular devices according to context information derived from received data, in accordance with at least one aspect of the present disclosure. [Figure 54B] FIG. 10 is a logic flow diagram of a process for controlling a second modular device according to context information derived from pre- and post-operative data received from a first modular device, in accordance with at least one aspect of the present disclosure. [Figure 54C] FIG. 10 is a logic flow diagram of a process for controlling a second modular device according to context information derived from pre- and post-operative data received from a first modular device and a second modular device, in accordance with at least one aspect of the present disclosure. [Figure 54D] FIG. 10 is a logic flow diagram of a process for controlling a third modular device according to context information derived from pre- and post-operative data received from a first modular device and a second modular device, in accordance with at least one aspect of the present disclosure. [Figure 55] FIG. 10 is a logic flow diagram for tracking data associated with operating room events in accordance with at least one aspect of the present disclosure. [Figure 56] FIG. 1 is a schematic diagram of a robotic surgical system during a surgical procedure including multiple hubs and an interactive secondary display, in accordance with at least one aspect of the present disclosure. [Figure 57] FIG. 58 is a detailed view of the interactive secondary display of FIG. 57 according to at least one embodiment of the present disclosure. [Figure 58] FIG. 10 is a diagram of pairing a personally owned wireless device with a surgical hub, according to at least one aspect of the present disclosure. [Figure 59] FIG. 1 is a diagram of an exemplary operating room (OR) setup, in accordance with at least one aspect of the present disclosure. [Figure 60] FIG. 10 is a logic flow diagram of a process for visually assessing surgical staff members in accordance with at least one aspect of the present disclosure. [Figure 61]1A-1C illustrate a series of models of surgical staff members during the course of a surgical procedure, in accordance with at least one aspect of the present disclosure. [Figure 62] 62 is a graph illustrating measured postures of the surgical staff member shown in FIG. 61 over time, in accordance with at least one embodiment of the present disclosure. [Figure 63] FIG. 1 illustrates a surgeon holding a surgical instrument, according to at least one embodiment of the present disclosure. [Figure 64] 10 is a scatter plot of wrist angle versus surgical outcome, in accordance with at least one aspect of the present disclosure. [Figure 65A] FIG. 10 is a logic flow diagram of a process for controlling a surgical device according to at least one aspect of the present disclosure. [Figure 65B] FIG. 1 is a logical flow diagram of a process for generating surgical metadata in accordance with at least one aspect of the present disclosure. [Figure 66] FIG. 1 is a block diagram of a gesture recognition system according to at least one aspect of the present disclosure. [Figure 67] FIG. 10 is a logic flow diagram of a process for controlling a display using situational awareness of a medical instrument. [Figure 68] 10A-10C illustrate one or more displays that may be controlled using situational awareness of one or more medical instruments during the course of a surgical procedure. [Figure 69] FIG. 1 is a logical flow diagram of a process for controlling a display using situational awareness to prioritize data displayed to a user. [Figure 70] FIG. 1 is a logical flow diagram of a process for displaying information on a display based on a user's visual focus. [Figure 71] A diagram illustrating one or more displays that may display information based on a user's visual focus is shown. [Figure 72] A diagram illustrating one or more displays that may display information based on the visual focus of one or more users is shown. [Figure 73]FIG. 1 is a logical flow diagram of a process for configuring data being displayed on a display. [Figure 74] FIG. 1 is a logic flow diagram of a process for controlling a display that may be outside the sterile field. DETAILED DESCRIPTION OF THE INVENTION

[0051] The applicant of the present application owns the following contemporaneously filed U.S. patent applications, each of which is hereby incorporated by reference in its entirety: · U.S. Patent Application No. 16 / 209,416, filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS"; · U.S. Patent Application No. 15 / 940671, entitled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER," filed March 29, 2018 (Attorney Docket No. END8502USNP); · U.S. Patent Application No. 16 / 182,269, entitled “IMAGE CAPTURING THE AREAS OUTSIDE THE ABDOMINAL TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE,” filed November 6, 2018 (Attorney Docket No.: END9018USNP3); · U.S. Patent Application No. 16 / 729,747, entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS," filed December 31, 2019 (Attorney Docket No. END9217USNP1); · U.S. Patent Application No. 16 / 729,778, entitled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE," filed December 31, 2019 (Attorney Docket No.: END9219USNP1); · U.S. Patent Application No. 16 / 729,807, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY," filed December 31, 2019 (Attorney Docket No. END9228USNP1); · U.S. Patent Application No. 15 / 940654, entitled "SURGICAL HUB SITUATIONAL AWARENESS," filed March 29, 2018 (Attorney Docket No. END8501USNP); · U.S. Patent Application No. 15 / 940,704, entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT," filed March 29, 2018 (Attorney Docket No. END8504USNP); · U.S. Patent Application No. 16 / 182,290, entitled "SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION," filed November 6, 2018 (Attorney Docket No. END9018USNP5); · U.S. Patent No. 9,011,427, entitled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued April 21, 2015; · U.S. Patent No. 9,123,155, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES," issued September 1, 2015; · U.S. Patent Application No. 16 / 209,478, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE," filed December 4, 2018 (Attorney Docket No. END9015USNP1); · U.S. Patent Application No. 16 / 182,246, entitled "ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES," filed November 6, 2018 (Attorney Docket No. END9016USNP1); · U.S. Patent Application No. 16 / 209,385, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (Attorney Docket No. END8495USNP); · U.S. Patent Application No. 16 / 209,407, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (Attorney Docket No. END8497USNP); · U.S. Patent Application No. 16 / 182,231, entitled “WIRELESS PAIRING OF A SURGICAL DEVICE WITH ANOTHER DEVICE WITHIN A STERILE SURGICAL FIELD BASED ON THE USAGE AND SITUATIONAL AWARENESS OF DEVICES,” filed November 6, 2018 (Attorney Docket No. END9032USNP2); · U.S. Patent Application No. 16 / 209,490, entitled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION," filed December 4, 2018 (Attorney Docket No. END9017USNP1); · U.S. Patent Application Publication No. 2014 / 0263552, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," published September 18, 2014; · U.S. Patent Application No. 15 / 628,175, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," filed June 20, 2017 (Attorney Docket No. END8199USNP); U.S. Patent Application Publication No. 2009 / 0046146, entitled "SURGICAL COMMUNICATION AND CONTROL SYSTEM," published February 19, 2009; and · U.S. Patent No. 9,283,054, entitled "SURGICAL APPARATUS WITH INDICATOR," issued on March 15, 2016.

[0052] 1 , a computer-implemented interactive surgical system 100 may include one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 that may include a remote server 113 coupled to a storage device). Each surgical system 102 may include at least one surgical hub 106 in communication with the cloud 104, which may include the remote server 113. In one example, as shown in FIG. 1 , a surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some embodiments, a surgical system 102 may include M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P may be integers greater than or equal to 1.

[0053] In various aspects, the visualization system 108 may include one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in FIG. 2. In one aspect, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described in U.S. Patent Application Publication No. 2019-0200844 A1, filed December 4, 2018, and entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0054] As shown in FIG. 2 , primary display 119 is positioned in the sterile field so as to be visible to the operator of operating table 114. In addition, visualization tower 111 is positioned outside the sterile field. Visualization tower 111 may include first non-sterile display 107 and second non-sterile display 109, which face away from each other. Visualization system 108, guided by hub 106, is configured to utilize displays 107, 109, and 119 to coordinate information flow to operators inside and outside the sterile field. For example, hub 106 may cause visualization system 108 to display snapshots of the surgical site as recorded by imaging device 124 on non-sterile display 107 or 109 while maintaining a live video of the surgical site on primary display 119. The snapshots on non-sterile display 107 or 109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0055] In one aspect, the hub 106 is also configured to send diagnostic input or feedback entered by the non-sterile operator at the visualization tower 111 to the primary display 119 in the sterile field for viewing by the sterile operator at the operating table. In one example, the input can be in the form of modifications to a snapshot displayed on the non-sterile display 107 or 109 and can be sent by the hub 106 to the primary display 119.

[0056] 2 , a surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 can also be configured to coordinate information flow to the display of the surgical instrument 112. See, for example, U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 111 can be transmitted by the hub 106 to the surgical instrument display 115 in the sterile field and viewed by the operator of the surgical instrument 112. Exemplary surgical instruments suitable for use with the surgical system 102 are described, for example, under the heading "Surgical Instrument Hardware" and in U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, and entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0057] FIG. 2 shows an example of a surgical system 102 being used to perform a surgical procedure on a patient lying on an operating table 114 in an operating room 116. A robotic system 110 may be used in the surgical procedure as part of the surgical system 102. The robotic system 110 may include a surgeon's console 118, a patient side cart 120 (surgical robot), and a surgical robot hub 122. The patient side cart 120 can manipulate at least one detachably coupled surgical tool 117 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 118. Images of the surgical site are acquired by a medical imaging device 124, which can be manipulated by the patient side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process and then display the images of the surgical site to the surgeon through the surgeon's console 118.

[0058] Other types of robotic systems can be readily adapted for use with the surgical system 102. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1), entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,407), the disclosure of which is incorporated herein by reference in its entirety.

[0059] Various examples of cloud-based analytics performed by the cloud 104 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1), entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,403), the disclosure of which is incorporated herein by reference in its entirety.

[0060] In various embodiments, the image capture device 124 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0061] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0062] The one or more illumination sources may be configured to emit electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and is sometimes referred to as visible light or simply light. The typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0063] The invisible spectrum (i.e., the non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0064] In various aspects, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngological-nephroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0065] Imaging devices can employ multispectral monitoring to determine topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging can extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed December 4, 2018 (U.S. Patent Application No. 16 / 209,385), the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical procedure is complete to perform one or more of the above-mentioned tests on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical equipment is necessary in any surgical procedure. The strict hygiene and sterilization conditions required in the "surgical field," i.e., the operating room or procedure room, require the highest possible sterility of all medical devices and equipment. Part of the sterilization process described above includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It is understood that the sterile field can be considered a specific area deemed free of microorganisms, such as in a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding the patient prepared for surgery. The sterile field can include cleaned team members wearing appropriate clothing, as well as all equipment and fixtures within the area.

[0066] Referring now to FIG. 3 , a hub 106 is shown in communication with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communications module 130, a processor module 132, a memory array 134, and an operating room mapping module 133. In certain embodiments, as shown in FIG. 3 , the hub 106 further includes a smoke evacuation module 126 and / or a suction / irrigation module 128. During a surgical procedure, the application of energy to tissue for sealing and / or cutting is commonly associated with smoke evacuation, the aspiration of excess fluid, and / or irrigation of tissue. Fluid, power, and / or data lines from different sources often become tangled during a surgical procedure. Addressing this issue can result in valuable time being lost during a surgical procedure. Untangling the lines may require unplugging them from their corresponding modules, which may require resetting the modules. The hub's modular enclosure 136 provides a unified environment for managing power, data, and fluid lines, reducing the frequency of tangling between such lines. An embodiment of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within the hub enclosure's docking station. The docking station includes data and power contacts. The combination generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component housed within a single unit. In one embodiment, the combination generator module also includes a smoke evacuation component, at least one energy delivery cable for connecting the combination generator module to a surgical instrument, at least one smoke evacuation component configured to evacuate smoke, fluid, and / or particulates generated by the application of therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke evacuation component.In one aspect, the fluid line is a first fluid line, and a second fluid line extends from a remote surgical site to an aspiration and irrigation module slidably received within the hub enclosure. In one aspect, the hub enclosure includes a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another, different energy type may be more beneficial for sealing tissue. For example, a bipolar generator may be used to seal tissue, while an ultrasonic generator may be used to cut the sealed tissue. Aspects of the present disclosure present a solution in which the hub's modular enclosure 136 is configured to house and facilitate interactive communication between various generators. One advantage of the hub's modular enclosure 136 is that it allows for rapid removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue and a first docking station including a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable into electrical engagement with the power and data contacts and the first energy generator module is slidably movable out of electrical engagement with the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station including a second docking port including second data and second power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power and data contacts and the second energy generator module is slidably movable out of electrical engagement with the second power and second data contacts.In addition, the modular surgical enclosure also includes a communication bus between the first and second docking ports configured to facilitate communication between the first and second energy generator modules. Referring to FIG. 3 , an aspect of the present disclosure is presented regarding a hub modular enclosure 136 that enables modular integration of a generator module 140, a smoke evacuation module 126, and a suction / irrigation module 128. The hub modular enclosure 136 further facilitates interactive communication between the modules 140, 126, and 128. The generator module 140 may be a generator module with integrated monopolar, bipolar, and ultrasonic components supported within a single housing unit slidably insertable into the hub modular enclosure 136. The generator module 140 may be configured to connect to a monopolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of monopolar, bipolar, and / or ultrasonic generator modules that interact via the hub modular enclosure 136. The hub modular enclosure 136 may be configured to facilitate insertion of multiple generators and interactive communication between the generators docked to the hub modular enclosure 136 such that the multiple generators function as a single generator.

[0067] FIG. 4 illustrates a surgical data network 201 comprising a modular communications hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped for surgical procedures, to a cloud-based system (e.g., a cloud 204 that may include a remote server 213 coupled to a storage device). In one aspect, the modular communications hub 203 comprises a network hub 207 and / or a network switch 209 in communication with a network router. The modular communications hub 203 can also be coupled to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, allowing data to travel from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network includes additional features that allow traffic to pass through the monitored surgical data network and configure each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0068] Modular devices 1a-1n located in an operating room may be coupled to modular communication hub 203. Network hub 207 and / or network switch 209 may be coupled to network router 211 to connect devices 1a-1n to cloud 204 or local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to local computer system 210 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be coupled to network switch 209. Network switch 209 may be coupled to network hub 207 and / or network router 211 to connect devices 2a-2m to cloud 204. Data associated with devices 2a-2n may be transferred to cloud 204 via network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transferred to local computer system 210 for local data processing and manipulation.

[0069] It will be appreciated that surgical data network 201 may be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. Modular communications hub 203 may be housed within a modular control tower configured to receive multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. Modular communications hub 203 is connected to a display 212 to display images acquired by some of devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include various modules such as, for example, an imaging module 138 coupled to an endoscope, a generator module 140 coupled to an energy-based surgical device, a smoke evacuation module 126, a suction / irrigation module 128, a communications module 130, a processor module 132, a storage array 134, a surgical device coupled to a display, and / or a non-contact sensor module, among other modular devices that may be connected to a modular communications hub 203 of a surgical data network 201.

[0070] In one aspect, the surgical data network 201 may include a combination of network hubs, network switches, and network routers that connect the devices 1a-1n / 2a-2m to the cloud. Any one or all of the devices 1a-1n / 2a-2m coupled to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. While the term "cloud" may be used as a metaphor for the "Internet," the term is not so limited. Accordingly, the term "cloud computing" may be used herein to refer to "a type of Internet-based computing" in which various services, such as servers, storage, and applications, are delivered via the Internet to a modular communications hub 203 and / or computer system 210 located at the surgical site (e.g., a fixed, mobile, temporary, or on-site operating room or space) and to devices connected to the modular communications hub 203 and / or computer system 210. The cloud infrastructure may be maintained by a cloud service provider. In this context, a cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located in the operating room. Hub hardware allows multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0071] By applying cloud computer data processing technology to data collected by the devices 1a-1n / 2a-2m, a surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. After tissue sealing and cutting procedures, at least some of the devices 1a-1n / 2a-2m can be used to observe tissue status and evaluate leakage or perfusion of the sealed tissue. Using cloud-based computing, at least some of the devices 1a-1n / 2a-2m can be used to diagnostically examine data, including images of body tissue samples, to identify pathologies, such as the effects of disease. Such data can include tissue localization and margin confirmation, as well as phenotyping. At least some of the devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with the imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by the devices 1a-1n / 2a-2m, including image data, can be transferred to the cloud 204 or a local computer system 210, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcome of the surgical procedure by determining whether further treatments can be performed, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions. Such data analysis may further employ prognostic analysis processes, and the use of standardized techniques can provide useful feedback to either confirm or suggest modifications to surgical treatment and surgeon performance.

[0072] The operating room devices 1a-1n may be connected to the modular communications hub 203 via wired or wireless channels, depending on the configuration of the devices 1a-1n relative to the network hub. The network hub 207, in one aspect, may be implemented as a local network broadcasting device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 207 can collect data in the form of packets and send them to a router in half-duplex mode. The network hub 207 may not store any media access control / Internet Protocol (MAC / IP) information for forwarding any device data. Only one of the devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 may not have a routing table or intelligence regarding where to send the information; it broadcasts all network data across each connection and to a remote server 213 (FIG. 4) on the cloud 204. Although network hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports can pose a security risk and cause bottlenecks.

[0073] The operating room devices 2a-2m may be connected to the network switch 209 via wired or wireless channels. The network switch 209 functions within the data link layer of the OSI model. The network switch 209 may be a multicast device for connecting the devices 2a-2m located in the same operating room to the network. The network switch 209 can transmit data in the form of frames to the network router 211 and functions in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through the network switch 209. The network switch 209 stores and uses the MAC addresses of the devices 2a-2m to forward data.

[0074] The network hub 207 and / or the network switch 209 may be coupled to a network router 211 to connect to the cloud 204. The network router 211 functions within the network layer of the OSI model. The network router 211 creates a route for transmitting data packets received from the network hub 207 and / or the network switch 211 to cloud-based computer resources for further processing and manipulation of data collected by any one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, such as different operating rooms in the same medical facility or different operating rooms in different medical facilities. The network router 211 can transmit data in the form of packets to the cloud 204 and functions in full-duplex mode. Multiple devices can transmit data simultaneously. The network router 211 uses IP addresses to forward data.

[0075] In one example, the network hub 207 may be implemented as a USB hub that allows multiple USB devices to be connected to a host computer. The USB hub can expand a single USB port into several tiers so that more ports are available for connecting devices to the host system computer. The network hub 207 may include wired or wireless capabilities for receiving information via wired or wireless channels. In one aspect, a wireless USB short-range, high-bandwidth wireless communication protocol may be used for communication between the devices 1a-1n and 2a-2m located in the operating room.

[0076] In an example, operating room devices 1a-1n / 2a-2m can communicate with modular communication hub 203 via Bluetooth wireless technology standard to exchange data over short distances from fixed and mobile devices (using short wavelength UHF radio waves in the ISM band of 2.4-2.485 GHz) and to create a personal area network (PAN). The operating room devices 1a-1n / 2a-2m can communicate with the modular communications hub 203 via numerous wireless or wired communications standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long Term Evolution (LTE), and any other wireless and wired protocols designated as EV-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as 3G, 4G, 5G, and beyond. The computing module may include multiple communications modules. For example, a first communications module may be dedicated to shorter-range wireless communications, such as Wi-Fi and Bluetooth, and a second communications module may be dedicated to longer-range wireless communications, such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and EV-DO.

[0077] The modular communications hub 203 may act as a central connection for one or all of the operating room devices 1a-1n / 2a-2m and may handle data types known as frames. The frames may carry data generated by the devices 1a-1n / 2a-2m. Once the frames are received by the modular communications hub 203, they are amplified and transmitted to the network router 211, which forwards this data to cloud computing resources using a number of wireless or wired communications standards or protocols, as described herein.

[0078] The modular communications hub 203 may be used as a stand-alone device or may be connected to compatible network hubs and network switches to form a larger network. Because the modular communications hub 203 is generally easy to install, configure, and maintain, the modular communications hub 203 may be a good choice for networking the operating room devices 1a-1n / 2a-2m.

[0079] FIG. 5 illustrates a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to the surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204, which may include a remote server 213. In one aspect, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to multiple operating room devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in FIG. 6, the modular control tower 236 includes a modular communication hub 203 coupled to a computer system 210.

[0080] As shown in the example of FIG. 5 , modular control tower 236 may be coupled to an imaging module 238 coupled to endoscope 239, a generator module 240 which may be coupled to energy device 241, a smoke evacuation module 226, a suction / irrigation module 228, a communications module 230, a processor module 232, a storage array 234, a smart device / instrument 235 which may optionally be coupled to a display 237, and a non-contact sensor module 242. Operating room equipment may be coupled to cloud computing resources and data storage via modular control tower 236. Robotic hub 222 may also be connected to modular control tower 236 and cloud computing resources. Devices / instruments 235, visualization system 208, among others, may be coupled to modular control tower 236 via wired or wireless communication standards or protocols as described herein. Modular control tower 236 may be coupled to a hub display 215 (e.g., monitor, screen) for displaying and overlaying images received from the imaging module, device / instrument display, and / or other visualization system 208. The hub display may also display data received from devices connected to the modular control tower along with the images and overlaid images.

[0081] FIG. 6 illustrates a surgical hub 206 comprising multiple modules coupled to a modular control tower 236. The modular control tower 236 may comprise a modular communications hub 203, e.g., a network-connected device, and a computer system 210, e.g., for local processing, visualization, and imaging. As shown in FIG. 6, the modular communications hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communications hub 203 and transfer data associated with the modules to the computer system 210, cloud computing resources, or both. As shown in FIG. 6, each of the network hubs / switches in the modular communications hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and a local display 217. Communication to the cloud 204 may occur via either a wired or wireless communication channel.

[0082] The surgical hub 206 may use the non-contact sensor module 242 to measure the dimensions of the operating room and generate a map of the surgical site using either an ultrasonic non-contact measurement device or a laser-based non-contact measurement device. The ultrasonic-based non-contact sensor module may scan the operating room by transmitting bursts of ultrasound and receiving echoes as they bounce off the perimeter walls of the operating room, as described in U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety, and the sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module may, for example, scan the operating room by transmitting laser light pulses, receive laser light pulses that reflect off the exterior walls of the operating room, and compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0083] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be coupled to a communication module 247, storage 248, memory 249, non-volatile memory 250, and an input / output interface 251 via a system bus. The system bus may be any of several types of bus structure(s), including a memory bus or memory controller, a peripheral or external bus, and / or a local bus, using any of a variety of bus architectures, including, but not limited to, a 9-bit bus, an Industrial Standard Architecture (ISA), a Micro-Channel Architecture (MSA), an Extended ISA (EISA), an Intelligent Drive Electronics (IDE), a VESA Local Bus (VLB), a Peripheral Component Interconnect (PCI), a USB, an Advanced Graphics Port (AGP), a Personal Computer Memory Card International Association bus (PCMCIA), a Small Computer Systems Interface (SCSI), or any other specialized bus.

[0084] Processor 244 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In one aspect, the processor may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. The processor core includes 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory at up to 40MHz, a prefetch buffer to improve performance above 40MHz, 32KB of single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB of electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analog, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, more details of which are available in the product datasheet.

[0085] In one aspect, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller may be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0086] System memory includes both volatile and nonvolatile memory. The basic input / output system (BIOS), containing the basic routines for transferring information between elements within a computer system, such as during start-up, is stored in nonvolatile memory. For example, nonvolatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random-access memory (RAM), which acts as external cache memory. RAM is available in many forms, including static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM).

[0087] The computer system 210 may also include removable / non-removable, volatile / non-volatile computer storage media, such as disk storage devices. Disk storage devices may include, but are not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage devices may include the above storage media, either individually or in combination with other storage media. Other storage media include, but are not limited to, optical disk drives such as compact disc ROM (CD-ROM), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM (DVD-ROM) drives. Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

[0088] It should be understood that computer system 210 may include software that acts as an intermediary between users and the basic computer resources described in a suitable operating environment. Such software may include an operating system. The operating system, which may be stored on disk storage, may function to control and allocate resources of the computer system. System applications may take advantage of resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein may be implemented with various operating systems or combinations of operating systems.

[0089] A user may input commands or information into computer system 210 through input devices coupled to I / O interface 251. Input devices include, but are not limited to, pointing devices such as a mouse, trackball, stylus, or touchpad; keyboards; microphones; joysticks; gamepads; satellite dishes; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor through the system bus via interface ports. Interface ports include, for example, serial ports, parallel ports, game ports, and USB. Output devices use some of the same types of ports as input devices. Thus, for example, a USB port may be used to provide input to the computer system and to output information from the computer system to an output device. Output adapters may be provided to illustrate that there may be several output devices, such as monitors, displays, speakers, and printers, among other output devices that may require special adapters. Output adapters include, by way of example and not limitation, video and sound cards that provide a means of connection between an output device and the system bus. It should be noted that other devices and / or systems of devices, such as remote computers, may provide both input and output capabilities.

[0090] The computer system 210 can operate in a networked environment using logical connections to one or more remote or local computers, such as a cloud computer. The remote cloud computer can be a personal computer, server, router, network PC, workstation, microprocessor-based appliance, peer device, or other common network node, but typically includes many or all of the elements described with respect to a computer system. For simplicity, only memory storage devices are shown along with the remote computer. The remote computer may be logically connected to the computer system through a network interface, which may then be physically connected through a communications connection. The network interface may encompass communications networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, and the like. WAN technologies include, but are not limited to, point-to-point links, circuit-switched networks such as Integrated Services Digital Networks (ISDN) and its variants, packet-switched networks, and Digital Subscriber Lines (DSL).

[0091] In various embodiments, the computer system 210 of FIG. 6 , the imaging module 238 of FIG. 5 and FIG. 6 , and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any specialized digital signal processor (DSP) used to process digital images. The image processor may employ parallel computing using single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) techniques to increase speed and efficiency. The digital image processing engine may perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0092] The communications connection may refer to the hardware / software used to connect the network interface to the bus. While the communications connection is shown internal to the computer system for clarity of illustration, the communications connection may also be external to computer system 210. By way of example only, the hardware / software required to connect to the network interface may include internal and external technologies such as regular telephone-grade modems, modems including cable modems and DSL modems, ISDN adapters, and Ethernet cards.

[0093] FIG. 7 shows a logic diagram of a surgical instrument or tool control system 470 according to one or more embodiments of the present disclosure. The system 470 may include control circuitry. The control circuitry may include a microcontroller 461 with a processor 462 and a memory 468. For example, one or more of sensors 472, 474, 476 provide real-time feedback to the processor 462. A motor 482, driven by a motor driver 492, operably couples a longitudinally movable displacement member to drive the I-beam knife element. A tracking system 480 may be configured to determine the position of the longitudinally movable displacement member. The position information may be provided to the processor 462, which may be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the firing member, firing bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control I-beam firing, closure tube movement, shaft rotation, and articulation. The display 473 can display various operating conditions of the instrument and may include touch screen functionality for data entry. Information displayed on the display 473 can be overlaid with images acquired via the endoscopic imaging module.

[0094] In one embodiment, microcontroller 461 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In one embodiment, main microcontroller 461 may be, for example, an LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, including on-chip memory of 256 KB of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2 KB of EEPROM, one or more PWM modules, one or more QEI analog, and / or one or more 12-bit ADCs with 12 analog input channels, details of which are available in the product datasheet.

[0095] In one embodiment, the microcontroller 461 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also known under the trade name Hercules ARM Cortex R4, manufactured by Texas Instruments. The safety controller may be specifically configured for IEC 61508 and ISO 26262 safety limit applications, among others, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0096] The microcontroller 461 may be programmed to perform various functions, such as precise control over the speed and position of the knife and articulation system. In one embodiment, the microcontroller 461 may include a processor 462 and memory 468. The electric motor 482 may be a brushed direct current (DC) motor with a gearbox and mechanical linkage to the articulation or knife system. In one embodiment, the motor driver 492 may be an A3941 available from Allegro Microsystems, Inc. Other motor drivers may be easily substituted for use in the tracking system 480 with an absolute positioning system. A detailed description of absolute positioning systems is provided in U.S. Patent Application Publication No. 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety.

[0097] The microcontroller 461 may be programmed to provide precise control over the velocity and position of the displacement members and articulation system. The microcontroller 461 may be configured to calculate a response within the microcontroller 461 software. The calculated response may be compared to the measured response of the actual system to obtain an "observed" response, which may be used to determine actual feedback. The observed response may be a suitably adjusted value that balances the smooth, continuous nature of the simulated response with the measured response, which can detect external influences on the system.

[0098] In some examples, the motor 482 may be controlled by a motor driver 492 and may be used by the surgical instrument or tool firing system. In various forms, the motor 482 may be a brushed DC drive motor having a maximum rotational speed of, for example, about 25,000 RPM. In some examples, the motor 482 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 492 may include, for example, an H-bridge driver including field-effect transistors (FETs). The motor 482 may be powered by a power supply assembly releasably attached to the handle assembly or tool housing to provide control power to the surgical instrument or tool. The power supply assembly may include a battery, which may include multiple battery cells connected in series, that may be used as a power source to power the surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be a lithium-ion battery, which may be connectable to and separable from the power supply assembly.

[0099] The motor driver 492 may be the A3941 available from Allegro Microsystems, Inc. The A3941 492 may be a full-bridge controller for use with external N-channel power metal-oxide semiconductor field-effect transistors (MOSFETs), specifically designed for inductive loads such as brushed DC motors. The driver 492 may include an intrinsic charge pump regulator, which may provide full (>10V) gate drive for battery voltages up to 7V, allowing the A3941 to operate with reduced gate drive down to 5.5V. A bootstrap capacitor may be used to provide the required above-battery supply voltage for the N-channel MOSFETs. An internal charge pump for the high-side drive allows DC (100% duty cycle) operation. The full-bridge may be driven in fast or slow decay mode using diode or synchronous rectification. In slow decay mode, current recirculation is possible through either the high-side or low-side FET. The power FETs may be protected from shoot-through by a resistor-adjustable dead time. Integrated diagnostics indicate undervoltage, overtemperature, and power bridge faults and can be configured to protect the power MOSFETs under most short circuit conditions. Other motor drivers can be easily substituted for use in tracking system 480 with an absolute positioning system.

[0100] The tracking system 480 may include a controlled motor drive circuit arrangement including a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for an absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member including a rack of drive teeth for meshing engagement with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a firing member that may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a firing bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may be used generally to refer to any movable member of a surgical instrument or tool, such as a drive member, firing member, firing bar, I-beam, or any element that can be displaced. In one aspect, a longitudinally movable drive member may be coupled to a firing member, firing bar, and I-beam. Thus, the absolute positioning system may, in effect, track the linear displacement of an I-beam by tracking the linear displacement of the longitudinally movable drive member. In various aspects, the displacement member may be coupled to any suitable position sensor 472 for measuring linear displacement. Thus, the longitudinally movable drive member, firing member, firing bar, or I-beam, or combinations thereof, may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include a contact displacement sensor or a non-contact displacement sensor.The linear displacement sensor may include a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a magnetic sensing system comprising a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0101] The electric motor 482 may include a rotatable shaft operably interfaced with a gear assembly mounted in meshing engagement with a set of drive teeth or rack on the displacement member. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 472 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor arrangement may be connected to a linear actuator by a rack and pinion arrangement or to a rotary actuator by a spur gear or other connection. A power source may provide power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member with a rack of drive teeth formed thereon for meshing engagement with a corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable firing member, a firing bar, an I-beam, or a combination thereof.

[0102] One revolution of the sensor element associated with position sensor 472 may correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance traveled by the displacement member from point "a" to point "b" after one revolution of the sensor element coupled to the displacement member. The sensor mechanism may be coupled via a gear reduction that results in the position sensor 472 completing one or more revolutions relative to the full stroke of the displacement member. The position sensor 472 may complete multiple revolutions relative to the full stroke of the displacement member.

[0103] A series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction to provide a unique position signal for two or more revolutions of the position sensor 472. The state of the switches may be fed back to the microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 of the sensor mechanism may comprise a magnetic sensor, an analog rotation sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0104] The position sensor 472 may comprise any number of magnetic sensing elements, such as magnetic sensors classified according to whether they measure the total magnetic field or a vector component of the magnetic field. The technologies used to produce both types of magnetic sensors may involve many aspects of physics and electronics. Technologies used to sense magnetic fields include search coils, fluxgates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, giant magnetoresistance, magnetic tunnel junctions, giant magnetoimpedance, magnetostrictive / piezoelectric composites, magnetodiodes, magnetotransistors, fiber optics, magneto-optics, and microelectromechanical systems-based magnetic sensors, among others.

[0105] In one aspect, the position sensor 472 of the tracking system 480 with an absolute positioning system may comprise a magnetic rotation absolute positioning system. The position sensor 472 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 472 interfaces with the microcontroller 461 to provide the absolute positioning system. The position sensor 472 may be a low-voltage, low-power component and includes four Hall-effect elements in an area of ​​the position sensor 472 that may be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on-chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement simple and efficient algorithms for calculating hyperbolic and trigonometric functions, requiring only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 461 via a standard serial communications interface, such as a serial peripheral interface (SPI) interface. The position sensor 472 may provide 12-bit or 14-bit resolution and may be an AS5055 chip provided in a small QFN 16-pin 4x4x0.85mm package.

[0106] A tracking system 480 with an absolute positioning system may include and / or be programmed to implement a feedback controller, such as a PID, state feedback, and adaptive controller. A power supply converts a signal from the feedback controller into a physical input to the system, in this case a voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors may include sensor arrangements such as those described in U.S. Pat. No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combination circuitry to combine the calculated response with the measured response using algorithms such as weighted averages and theoretical control loops that drive the calculated response towards the measured response. The calculated response of the physical system may take into account properties such as mass, inertia, viscous friction, and induced drag in order to predict what the state and output of the physical system will be given knowledge of the input.

[0107] The absolute positioning system can provide the absolute position of the displacement member upon power-up of the instrument without retracting or advancing the displacement member to a reset (zero or home) position, as may be required with conventional rotary encoders that simply count the number of forward or backward steps taken by the motor 482 to estimate the position of the device actuator, drive bar, knife, etc.

[0108] Sensor 474, such as a strain gauge or micro-strain gauge, can be configured to measure one or more parameters of the end effector, such as the amplitude of strain exerted on the anvil during clamping, which can be indicative of the closure force applied to the anvil. The measured strain can be converted to a digital signal and provided to processor 462. Instead of or in addition to sensor 474, a sensor 476, such as a load sensor, can measure the closure force applied to the anvil by the closure drive system. For example, sensor 476, such as a load sensor, can measure the firing force applied to the I-beam during the firing stroke of the surgical instrument or tool. The I-beam is configured to engage a wedge-shaped sled, which is configured to cam the staple driver upward and drive the staples into deforming contact with the anvil. The I-beam can also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 478 can be used to measure the current drawn by motor 482. The force required to advance the firing member may correspond, for example, to the current drawn by motor 482. The measured force may be converted to a digital signal and provided to processor 462.

[0109] In one form, a strain gauge sensor 474 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue grasped by the end effector can include a strain gauge sensor 474, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one aspect, the strain gauge sensor 474 can measure the amplitude or magnitude of strain exerted on the jaw members of the end effector during clamping, which can be indicative of tissue compression. The measured strain can be converted to a digital signal and provided to the processor 462 of the microcontroller 461. The load sensor 476 can measure the force used to operate the knife element, for example, to cut tissue captured between the anvil and the staple cartridge. A magnetic field sensor can be used to measure the thickness of the captured tissue. The magnetic field sensor measurements can also be converted to a digital signal and provided to the processor 462.

[0110] Measurements of tissue compression, tissue thickness, and / or force required to close the end effector on the tissue, measured by sensors 474, 476, respectively, can be used by microcontroller 461 to characterize a selected position of the firing member and / or a corresponding value of firing member velocity. In one example, memory 468 can store techniques, equations, and / or look-up tables that can be used by microcontroller 461 during evaluation.

[0111] The surgical instrument or tool control system 470 may also include wired or wireless communication circuitry for communicating with the modular communications hub 203 as shown in FIGS.

[0112] 8 illustrates a surgical instrument or tool with multiple motors that can be activated to perform various functions. In certain examples, a first motor can be activated to perform a first function, a second motor can be activated to perform a second function, a third motor can be activated to perform a third function, a fourth motor can be activated to perform a fourth function, and so on. In certain examples, the motors of the robotic surgical instrument 600 can be individually activated to produce firing, closing, and / or articulation motions in the end effector. The firing, closing, and / or articulation motions can be transmitted to the end effector via, for example, a shaft assembly.

[0113] In certain examples, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably coupled to a firing motor drive assembly 604, which may be configured to transfer the firing motion generated by the motor 602 to the end effector, particularly to displace an I-beam element. In certain examples, the firing motion generated by the motor 602 may, for example, deploy staples from a staple cartridge into tissue captured by the end effector and / or advance a cutting blade of the I-beam element to cut the captured tissue. The I-beam element may be retracted by reversing the direction of the motor 602.

[0114] In certain examples, the surgical instrument or tool may include a closure motor 603. The closure motor 603 may be operatively coupled to a closure motor drive assembly 605, which may be configured to transmit the closure motion generated by the motor 603 to the end effector, specifically to displace a closure tube to close the anvil and compress tissue between the anvil and the staple cartridge. The closure motion may transition the end effector from an open configuration to an approximation configuration, for example, to capture tissue. The end effector may be transitioned to the open position by reversing the direction of the motor 603.

[0115] In certain examples, a surgical instrument or tool may include, for example, one or more articulation motors 606 a, 606 b. The motors 606 a, 606 b may be operatively coupled to corresponding articulation motor drive assemblies 608 a, 608 b, which may be configured to transfer articulation motion generated by the motors 606 a, 606 b to an end effector. In certain examples, the articulation motion may, for example, cause the end effector to articulate relative to the shaft.

[0116] As described herein, a surgical instrument or tool may include multiple motors that can be configured to perform various independent functions. In certain examples, multiple motors of a surgical instrument or tool can be activated individually or separately to perform one or more functions while other motors remain stopped. For example, articulation motors 606 a, 606 b can be activated to articulate the end effector while firing motor 602 remains stopped. Alternatively, firing motor 602 can be activated to fire multiple staples and / or advance a cutting blade while articulation motor 606 remains stopped. Additionally, closure motor 603 can be activated simultaneously with firing motor 602 to distally advance a closure tube and I-beam element, as described in more detail herein below.

[0117] In certain examples, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In certain examples, the common control module 610 can accommodate one of the multiple motors at a time. For example, the common control module 610 may be individually connectable and detachable to multiple motors of a robotic surgical instrument. In certain examples, the multiple motors of a surgical instrument or tool may share one or more common control modules, such as the common control module 610. In certain examples, the multiple motors of a surgical instrument or tool can be individually and selectively engaged with the common control module 610. In certain examples, the common control module 610 can selectively switch from interfacing with one of the multiple motors of the surgical instrument or tool to interfacing with another of the multiple motors of the surgical instrument or tool.

[0118] In at least one example, common control module 610 can be selectively switched between operative engagement with articulation motors 606 a, 606 b and operative engagement with either firing motor 602 or closure motor 603. In at least one example, as shown in FIG. 8 , switch 614 can be moved or transitioned between multiple positions and / or states. For example, in a first position 616, switch 614 can electrically couple common control module 610 to firing motor 602, in a second position 617, switch 614 can electrically couple common control module 610 to closure motor 603, in a third position 618 a, for example, switch 614 can electrically couple common control module 610 to first articulation motor 606 a, and in a fourth position 618 b, switch 614 can electrically couple common control module 610 to second articulation motor 606 b. In certain examples, a separate common control module 610 may be electrically coupled to the firing motor 602, the closing motor 603, and the articulation motors 606 a, 606 b at the same time. In certain examples, the switch 614 may be a mechanical switch, an electromechanical switch, a solid-state switch, or any suitable switching mechanism.

[0119] Each of the motors 602, 603, 606a, 606b may be equipped with a torque sensor to measure the output torque on the shaft of the motor. The force on the end effector may be sensed in any conventional manner, such as by a force sensor outside the jaws or by a torque sensor on the motor that actuates the jaws.

[0120] 8, common control module 610 may include a motor driver 626, which may include one or more H-bridge FETs. Motor driver 626 may modulate power transferred from a power supply 628 to a motor coupled to common control module 610, for example, based on input from a microcontroller 620 ("controller"). In certain examples, microcontroller 620 may be used to determine, for example, the current drawn by a motor while the motor is coupled to common control module 610, as described herein.

[0121] In particular examples, microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-transitory computer-readable media or memory units 624 ("memory"). In particular examples, memory 624 may store various program instructions that, when executed, cause processor 622 to perform multiple functions and / or calculations described herein. In particular examples, one or more of memory units 624 may be coupled to processor 622, for example.

[0122] In certain examples, power supply 628 may be used to, for example, power microcontroller 620. In certain examples, power supply 628 may include a battery (or "battery pack" or "power pack"), such as, for example, a lithium-ion battery. In certain examples, the battery pack may be configured to be releasably attached to the handle to power surgical instrument 600. Several battery cells connected in series may be used as power supply 628. In certain examples, power supply 628 may be, for example, replaceable and / or rechargeable.

[0123] In various examples, the processor 622 can control the motor drivers 626 to control the position, direction of rotation, and / or speed of the motors coupled to the common control module 610. In certain examples, the processor 622 can signal the motor drivers 626 to stop and / or disable the motors coupled to the common control module 610. The term "processor," as used herein, should be understood to include any suitable microprocessor, microcontroller, or other basic computing device that integrates the functionality of a computer's central processing unit (CPU) on one integrated circuit or up to a few integrated circuits. A processor may be a general-purpose programmable device that accepts digital data as input, processes the data according to instructions stored in memory, and provides a result as output. Because it may have internal memory, it may be an example of sequential digital logic. A processor may operate on numbers and symbols represented in the binary system.

[0124] Processor 622 may be any single-core or multi-core processor, such as those known under the trade name ARM Cortex manufactured by Texas Instruments. In a particular example, microcontroller 620 may be, for example, the LM 4F230H5QR available from Texas Instruments. In at least one embodiment, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes, among other features readily available in the product datasheet, 256 KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40 MHz, a prefetch buffer to improve performance above 40 MHz, 32 KB of single-cycle SRAM, internal ROM loaded with StellarisWare® software, 2 KB of EEPROM, one or more PWM modules, one or more QEI analog, and one or more 12-bit ADCs with 12 analog input channels. Other microcontrollers may be readily substituted for use with module 4410. Accordingly, the present disclosure should not be limited in this context.

[0125] The memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600 that are connectable to the common control module 610. For example, the memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the articulation motors 606 a, 606 b. Such program instructions may cause the processor 622 to control the firing, closing, and articulation functions according to inputs from an algorithm or control program of the surgical instrument or tool.

[0126] For example, one or more mechanisms and / or sensors, such as sensor 630, can be used to alert processor 622 to program instructions to use in a particular setting. For example, sensor 630 can alert processor 622 to use program instructions associated with firing, closing, and articulating the end effector. In certain examples, sensor 630 can include a position sensor that can be used to sense the position of switch 614, for example. Thus, processor 622 can use program instructions associated with firing an I-beam of the end effector when it detects, for example, via sensor 630, that switch 614 is in first position 616; processor 622 can use program instructions associated with closing an anvil when it detects, for example, that switch 614 is in second position 617 via sensor 630; and processor 622 can use program instructions associated with articulating the end effector when it detects, for example, via sensor 630, that switch 614 is in third position 618a or fourth position 618b.

[0127] 9 shows a diagram of a context-aware surgical system 5100 in accordance with at least one aspect of the present disclosure. In some examples, the data sources 5126 may include, for example, the modular device 5102 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 5122 (e.g., an EMR database containing patient records), and patient monitoring devices 5124 (e.g., blood pressure (BP) monitors and electrocardiography (EKG) monitors). The surgical hub 5104 may be configured to derive contextual information regarding the surgical procedure from the data based, for example, on a particular combination of the received data or a particular order in which the data is received from the data sources 5126. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure the surgeon is performing, the type of tissue being operated on, or the body cavity being treated. This ability of some aspects of the surgical hub 5104 to derive or infer information regarding the surgical procedure from the received data may be referred to as “situational awareness.” In one example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104 that derives contextual information related to the surgical procedure from received data.

[0128] The situational awareness system of the surgical hub 5104 can be configured to derive contextual information from data received from the data sources 5126 in a variety of different ways. Illustratively, the situational awareness system may include a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained with training data to correlate various inputs (e.g., data from the database 5122, the patient monitor 5124, and / or the modular device 5102) with corresponding contextual information related to the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information related to the surgical procedure from provided inputs. In examples, the situational awareness system may include a lookup table that stores pre-characterized contextual information related to the surgical procedure in association with one or more inputs (or ranges of inputs) corresponding to the contextual information. In response to querying with one or more inputs, the lookup table can return corresponding contextual information for the situational awareness system to control the modular device 5102. In examples, the contextual information received by the situational awareness system of the surgical hub 5104 may be associated with a particular control adjustment or set of control adjustments for one or more modular devices 5102. In examples, the situational awareness system may include a further machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 5102 when provided with the contextual information as input.

[0129] A surgical hub 5104 incorporating a situational awareness system may provide several advantages to the surgical system 5100. One advantage may include improved interpretation of sensed and collected data, which improves processing accuracy and / or use of the data during the course of a surgical procedure. Returning to the previous example, the situational aware surgical hub 5104 may be able to determine what type of tissue is being operated on, and thus, if an unexpectedly high force to close the end effector of the surgical instrument is detected, the situational aware surgical hub 5104 may correctly accelerate or decelerate the motor of the surgical instrument to match the tissue type.

[0130] The type of tissue being operated on can affect the adjustments made to the compression speed and load threshold of the surgical stapling and severing instrument for a particular tissue gap measurement. The context-aware surgical hub 5104 can infer whether the surgical procedure being performed is thoracic or abdominal surgery, which enables the surgical hub 5104 to determine whether the tissue being clamped by the end effector of the surgical stapling and severing instrument is pulmonary (in the case of thoracic surgery) or stomach (in the case of abdominal surgery). The surgical hub 5104 can then adjust the compression speed and load threshold of the surgical stapling and severing instrument appropriately for the tissue type.

[0131] The type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situation-aware surgical hub 5104 can determine if the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the procedure type. Generally, certain procedure types can be performed within specific body cavities, so the surgical hub 5104 can control the smoke evacuator motor speed appropriately for the body cavity being operated on. Thus, the situation-aware surgical hub 5104 can provide a consistent amount of smoke evacuation for both thoracic and abdominal procedures.

[0132] The type of procedure being performed can affect the optimal energy level at which an ultrasonic surgical instrument or radio frequency (RF) electrosurgical instrument operates. For example, an arthroscopic procedure may require a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The context-aware surgical hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The context-aware surgical hub 5104 can determine which type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the tissue geometry expected for the surgical procedure. Additionally, the situation-aware surgical hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the course of a surgical procedure, rather than simply on a procedure-by-procedure basis. The situation-aware surgical hub 5104 can determine which step of the surgical procedure is being performed or will continue to be performed, and then update the generator and / or control algorithms of the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type following the surgical procedure step.

[0133] In examples, the surgical hub 5104 may derive data from additional data sources 5126 to improve conclusions drawn from one data source 5126. The situation-aware surgical hub 5104 may augment the data received from the modular device 5102 with contextual information constructed about the surgical procedure from other data sources 5126. For example, the situation-aware surgical hub 5104 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, the video or image data may be inconclusive. Thus, in one example, the surgical hub 5104 may be further configured to compare a physiological measurement (e.g., blood pressure sensed on a BP monitor communicatively connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (FIG. 2) communicatively coupled to the surgical hub 5104) to make a determination regarding the integrity of a staple line or tissue weld. In other words, the situational awareness system of the surgical hub 5104 can take physiological measurement data into account to provide additional context when analyzing the visualization data, which can be useful when the visualization data may not be conclusive or incomplete on its own.

[0134] For example, the situation-aware surgical hub 5104 can proactively activate a generator to which an RF electrosurgical instrument is connected if it is determined that a subsequent step in a procedure will require the use of the instrument. By proactively activating the energy source, the instrument can be ready for use as soon as the previous step in the procedure is completed.

[0135] The situation-aware surgical hub 5104 can determine whether the current or subsequent steps in the surgical procedure require different views or magnifications on the display according to the characteristics of the surgical site that the surgeon is expected to need to see. The surgical hub 5104 can then proactively change the displayed views (e.g., provided by medical imaging devices for the visualization system 108) appropriately, so that the display automatically adjusts throughout the surgical procedure.

[0136] The context-aware surgical hub 5104 can determine which step of the surgical procedure is being performed or will be performed next, and whether specific data or data comparisons are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the step of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0137] Errors may be checked during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situation-aware surgical hub 5104 may determine whether the surgical field is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 may be configured to determine the type of surgical procedure being performed, retrieve (e.g., from memory) the corresponding checklist, product locations, or setup requirements, and then compare the current surgical field layout to a standard layout for the type of surgical procedure the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 may be configured to compare the list of items for the procedure and / or the list of devices paired with the surgical hub 5104 to a recommended or expected manifest of items and / or devices for a given surgical procedure. If a discontinuity exists between the lists, the surgical hub 5104 may be configured to provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, and / or other surgical item is missing. In some examples, the surgical hub 5104 can be configured to determine the relative distance or relative position of the modular device 5102 and the patient monitoring device 5124, for example, by proximity sensors. The surgical hub 5104 can compare the relative positions of the devices to a recommended or predicted layout for a particular surgical procedure. If a discrepancy exists between the layouts, the surgical hub 5104 can be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0138] The context-aware surgical hub 5104 can determine whether a surgeon (or other medical personnel) is making an error or deviating from an expected sequence of actions during the course of a surgical procedure. For example, the surgical hub 5104 can be configured to determine the type of surgical procedure being performed, retrieve (e.g., from memory) a corresponding list of steps or sequences of equipment use, and then compare the steps being performed or the equipment being used during the course of the surgical procedure with the expected steps or equipment for the type of surgical procedure that the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 can be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0139] The surgical instruments (and other modular devices 5102) may be tailored to the specific circumstances of each surgical procedure (such as for different tissue types) and may verify actions during the surgical procedure. Subsequent steps, data, and display adjustments may be provided to the surgical instruments (and other modular devices 5102) within the surgical site according to the specific circumstances of the procedure.

[0140] 10 illustrates a timeline 5200 of an exemplary surgical procedure and the contextual information the surgical hub 5104 may derive from data received from the data sources 5126 at each step of the surgical procedure. The following description of the timeline 5200 shown in FIG. 9 also refers to FIG. 9. The timeline 5200 may illustrate the general steps that nurses, surgeons, and other medical personnel may take during the course of a lung segmentectomy surgery, beginning with the setup of the surgical site and concluding with the transport of the patient to the post-operative recovery room. The context-aware surgical hub 5104 may receive data from the data sources 5126 throughout the course of the surgical procedure, including data generated each time medical personnel use a modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular devices 5102 and other data sources 5126 and can continually derive inferences (i.e., contextual information) regarding the ongoing procedure as new data is received, such as which step of the procedure is occurring at any given time. The situational awareness system of the surgical hub 5104 may be able to, for example, record data regarding the procedure to generate reports, verify steps being taken by medical personnel, provide data or prompts (e.g., via a display screen) that may be relevant to particular procedure steps, adjust the modular device 5102 based on the context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and take any other such action described herein.

[0141] As a first step 5202 in this exemplary procedure, hospital personnel may retrieve the patient's EMR from the hospital's EMR database. Based on selected patient data in the EMR, the surgical hub 5104 determines that the procedure to be performed is thoracic surgery. In a second step 5204, the personnel may scan incoming medical supplies for the procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies that may be utilized in various types of procedures and verifies that the combination of supplies matches the thoracic procedure. Furthermore, the surgical hub 5104 may also determine that the procedure is not a wedge resection (either because the incoming supplies do not include specific supplies required for a thoracic wedge resection or are otherwise not compatible with a thoracic wedge resection). In a third step 5206, medical personnel may scan the patient band via a scanner 5128 communicatively connected to the surgical hub 5104. The surgical hub 5104 may then verify the patient's identity based on the scanned data. In a fourth step 5208, medical personnel turn on the auxiliary devices. The auxiliary devices utilized may vary according to the type of surgical procedure and the technology used by the surgeon, but in this exemplary case include a smoke evacuator, an aspirator, and a medical imaging device. Once activated, the auxiliary device, which is a modular device 5102, may automatically pair with the surgical hub 5104, which may be located within a particular proximity of the modular device 5102, as part of its initialization process. The surgical hub 5104 may then derive contextual information regarding the surgical procedure by detecting the type of modular device 5102 that is paired with it during this pre-operative or initialization phase. In this particular example, the surgical hub 5104 may determine that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the types of modular devices 5102 connecting to the hub, the surgical hub 5104 may roughly deduce the particular procedure the surgical team will be performing.Once the surgical hub 5104 knows what particular procedure is being performed, it can then retrieve the steps of that procedure from memory or from the cloud and then cross-reference data subsequently received from connected data sources 5126 (e.g., modular devices 5102 and patient monitors 5124) to deduce which steps of the surgical procedure the surgical team is performing. In a fifth step 5210, personnel attach EKG electrodes and other patient monitors 5124 to the patient. The EKG electrodes and other patient monitors 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitors 5124, the surgical hub 5104 can confirm that the patient is in the operating room, for example, as described in process 5207. In a sixth step 5212, medical personnel can administer anesthesia to the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from the modular device 5102 and / or the patient monitor 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Upon completion of the sixth step 5212, the pre-operative portion of the lung segmentectomy surgery is complete and the surgical portion begins.

[0142] In a seventh step 5214, the patient's lung being operated on may be collapsed (while ventilation is switched to the contralateral lung). The surgical hub 5104 may, for example, infer from ventilator data that the patient's lung has been collapsed. The surgical hub 5104 may compare the detection of the patient's lung being collapsed with the expected steps of the procedure (which may be accessed or retrieved in advance) and therefore infer that the surgical portion of the procedure has begun, thereby determining that collapsing the lung may be the first surgical step in this particular procedure. In an eighth step 5216, the medical imaging device 5108 (e.g., a scope) may be inserted and video from the medical imaging device may be initiated. The surgical hub 5104 may receive medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 may determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 5104 may determine that the particular procedure being performed is a segmentectomy as opposed to a lobectomy (note that a wedge resection has not already been taken into account by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Data from the medical imaging device 124 (FIG. 2) may be utilized to determine contextual information regarding the type of procedure being performed in various ways, such as by determining the angle of the medical imaging device pointed relative to the visualization of the patient's anatomy, by monitoring the number or medical imaging devices being utilized (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being utilized. For example, one technique for performing a VATS lobectomy may position the camera above the diaphragm in the anterior-inferior corner of the patient's thoracic cavity, while one technique for performing a VATS segmentectomy may position the camera in an intercostal position anterior to the segmental fissure. The situational awareness system may be trained to recognize the position of the medical imaging device according to the visualization of the patient's anatomy, for example, using pattern recognition or machine learning techniques. An exemplary technique for performing a VATS lobectomy may utilize a single medical imaging device.An exemplary technique for performing a VATS segmentectomy utilizes multiple cameras. One technique for performing a VATS segmentectomy utilizes an infrared light source (which may be communicatively coupled to the surgical hub as part of a visualization system) to visualize the segmental fissure, which is not utilized in a VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the particular type of surgical procedure being performed and / or the technique being used for the particular type of surgical procedure.

[0143] In a ninth step 5218, the surgical team may begin the incision step of the procedure. Because the surgical hub 5104 receives data from the RF or ultrasonic generator indicating that an energy instrument is being fired, it may infer that the surgeon is in the process of incising and separating the patient's lungs. The surgical hub 5104 may cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being fired at this point in the process (i.e., after the steps of the procedure described above have been completed) corresponds to the incision step. In a tenth step 5220, the surgical team may proceed to the ligation step of the procedure. Because the surgical hub 5104 may receive data from the surgical stapling and severing instrument indicating that the instrument is being fired, it may infer that the surgeon is ligating arteries and veins. As with the previous step, the surgical hub 5104 may derive this inference by cross-referencing the receipt of data from the surgical stapling and severing instrument with the steps in the retrieved process. In an eleventh step 5222, the segmentectomy portion of the procedure may be performed. Based on data from the surgical stapling and severing instrument (including data from its cartridge), the surgical hub 5104 can infer that the surgeon is transecting parenchyma. The cartridge data can correspond, for example, to the size or type of staples being fired by the instrument. Because different types of staples are applied to different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transected. In this case, the type of staples being fired is applied to parenchyma (or other similar tissue type), allowing the surgical hub 5104 to infer that the segmentectomy portion of the procedure is being performed. Subsequently, in a twelfth step 5224, a node dissection step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 5104 can infer that the surgical team is dissecting nodes and performing a leak test. In this particular procedure, the RF or ultrasonic instrument utilized after the parenchyma has been transected corresponds to the node dissection step, allowing the surgical hub 5104 to make this inference.It should be noted that surgeons will routinely alternate between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasonic) instruments depending on the particular step in the procedure, as different instruments are better suited for particular tasks. Thus, the particular sequence in which the stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Once the twelfth step 5224 is completed, the incision is closed and the post-operative portion of the procedure can begin.

[0144] In a thirteenth step 5226, the patient may be deanesthetized. The surgical hub 5104 may estimate that the patient is emerging from anesthesia, for example, based on ventilator data (i.e., the patient's breathing rate begins to increase). Finally, a fourteenth step 5228 may be a step in which medical personnel remove the various patient monitors 5124 from the patient. Thus, the surgical hub 5104 may estimate that the patient is being transferred to a recovery room when the hub loses EKG, BP, and other data from the patient monitors 5124. As can be seen from this exemplary procedure description, the surgical hub 5104 can determine or estimate when each step of a given surgical procedure is occurring according to data received from the various data sources 5126 communicatively coupled to the surgical hub 5104.

[0145] As shown in the first step 5202 of the timeline 5200 shown in FIG. 10 , in addition to utilizing patient data from the EMR database to estimate the type of surgical procedure to be performed, the patient data can also be utilized by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular devices 5102.

[0146] FIG. 11 is a block diagram of a computer-implemented interactive surgical system according to at least one embodiment of the present disclosure. In one embodiment, the computer-implemented interactive surgical system may be configured to monitor and analyze data related to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and surgical sites or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analysis system. While the cloud-based analysis system may be described as a surgical system, it is not necessarily limited to such and may also be a cloud-based medical system. As shown in FIG. 11 , the cloud-based analysis system may include a plurality of surgical instruments 7012 (which may be the same as or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same as or similar to hub 106), and a surgical data network 7001 (which may be the same as or similar to network 201) for coupling the surgical hubs 7006 to cloud 7004 (which may be the same as or similar to cloud 204). Each of the plurality of surgical hubs 7006 may be communicatively coupled to one or more surgical instruments 7012. The hub 7006 may also be communicatively coupled to a cloud 7004 of computer-implemented interactive surgical systems via a network 7001. The cloud 7004 may be a remote, centralized source of hardware and software for storing, manipulating, and communicating data generated based on the operation of various surgical systems. As shown in FIG. 11 , access to the cloud 7004 may be achieved via the network 7001, which may be the Internet or other suitable computer network. The surgical hub 7006, which may be coupled to the cloud 7004, may be considered the client side of a cloud computing system (i.e., a cloud-based analysis system). A surgical instrument 7012 may be paired with the surgical hub 7006 for control and implementation of the various surgical procedures or operations described herein.

[0147] Additionally, the surgical instrument 7012 may include a transceiver for transmitting data to and from a corresponding surgical hub 7006 (which may also include a transceiver). The combination of the surgical instrument 7012 and the corresponding hub 7006 can indicate a specific location, such as a surgical site within a medical facility (e.g., a hospital) for providing a medical procedure. For example, the memory of the surgical hub 7006 can store the location data. As shown in FIG. 11 , the cloud 7004 includes a central server 7013 (which may be the same as or similar to the remote server 7013), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7006. The central server 7013 of the cloud 7004 collectively manages the cloud computing system, which includes monitoring requests by the client modules 7006 and managing the processing power of the cloud 7004 to execute those requests. Each of the central servers 7013 may include one or more processors 7008 coupled to a suitable memory device 7010, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as a magnetic storage device. The memory device 7010 may include machine-executable instructions that, when executed, cause the processor 7008 to execute a data analysis module 7034 for cloud-based data analysis, actions, recommendations, and other operations described below. Further, the processor 7008 may execute the data analysis module 7034 independently or in conjunction with a hub application executed independently by the hub 7006. The central server 7013 may also include a database 2212 of aggregated medical data, which may reside in the memory 2210.

[0148] Based on its connection to the various surgical hubs 7006 via the network 7001, the cloud 7004 can aggregate data from the various surgical instruments 7012 and the particular data generated by their corresponding hubs 7006. Such aggregated data may be stored in an aggregated medical database 7011 of the cloud 7004. Specifically, the cloud 7004 can advantageously perform data analysis and operations on the aggregated data to provide insights and / or perform functions that an individual hub 7006 cannot accomplish on its own. To this end, as shown in FIG. 11 , the cloud 7004 and the surgical hubs 7006 are communicatively coupled to send and receive information. An I / O interface 7005 is connected to the multiple surgical hubs 7006 via the network 7001. In this manner, the I / O interface 7005 can be configured to transfer information between the surgical hubs 7006 and the database 7011 of aggregated medical data. Thus, the I / O interface 7005 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hub 7006. These requests may be sent to the hub 7006 via a hub application. The I / O interface 7005 may include one or more high-speed data ports, such as a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 7004 to the hub 7006. The hub application server 7002 of the cloud 7004 may be configured to host and provide shared functionality to software applications (e.g., hub applications) executed by the surgical hub 7006. For example, the hub application server 7002 may manage requests by the hub application through the hub 7006, control access to the database 7011 of aggregated medical data, and perform load balancing. The data analysis module 7034 is described in more detail with reference to FIG. 12 .

[0149] The particular cloud computing system configurations described in this disclosure can be specifically designed to address various problems that arise in the context of medical surgeries and procedures performed using medical devices, such as surgical instruments 7012, 112. In particular, the surgical instruments 7012 may be digital surgical devices configured to interact with the cloud 7004 to implement techniques for improving surgical outcomes. The various surgical instruments 7012 and / or the surgical hub 7006 may include touch-controlled user interfaces to allow a clinician to control aspects of the interaction between the surgical instruments 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, can also be used.

[0150] FIG. 12 is a block diagram illustrating the functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analysis system may include multiple data analysis modules 7034 that may be executed by a processor 7008 of a cloud 7004 to provide data analysis solutions to problems that arise specifically in the medical field. As shown in FIG. 12 , the functionality of the cloud-based data analysis modules 7034 may be supported via a hub application 7014 hosted by a hub application server 7002 that is accessible on a surgical hub 7006. The cloud processor 7008 and the hub application 7014 may work in conjunction to execute the data analysis modules 7034. An application program interface (API) 7016 may define a set of protocols and routines corresponding to the hub application 7014. Additionally, the API 7016 may manage the storage and retrieval of data to a centralized medical database 7011 for operation of the applications 7014. A cache 7018 may also store data (e.g., temporarily) and may be coupled to the API 7016 for more efficient retrieval of data used by the applications 7014. 12 may include modules for resource optimization 7020, data collection and aggregation 7022, authorization and security 7024, control program updates 7026, patient outcome analysis 7028, recommendations 7030, and data classification and prioritization 7032. Other suitable data analysis modules may also be implemented by the cloud 7004, according to some aspects. In one aspect, the data analysis module may be used to make specific recommendations based on an analysis of trends, outcomes, and other data.

[0151] For example, the data collection and aggregation module 7022 may be used to generate self-describing data (e.g., metadata), including identifying notable features or configurations (e.g., trends), managing redundant data sets, and storing data in paired data sets that may be grouped by procedure but not necessarily matched to actual surgical procedure dates and surgeons. In particular, paired data sets generated from the operation of the surgical instrument 7012 may include applying a binary classification, such as a bleeding or non-bleeding event. More generally, the binary classification may be characterized as either a desired event (e.g., a successful surgical procedure) or an undesired event (e.g., a misfired or misused surgical instrument 7012). The aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Thus, the data collection and aggregation module 7022 may generate aggregated metadata or other organized data based on the raw data received from the surgical hub 7006. To this end, the processor 7008 may be operatively coupled to the hub application 7014 and the database of aggregated medical data 7011 for executing the data analysis module 7034. The data collection and aggregation module 7022 may store the aggregated, organized data in the database of aggregated medical data 2212.

[0152] The resource optimization module 7020 can be configured to analyze this aggregated data to determine optimal use of resources for a particular medical facility or group of medical facilities. For example, the resource optimization module 7020 can determine optimal order points for surgical instruments 7012 for a group of medical facilities based on corresponding predicted demand for such surgical instruments 7012. The resource optimization module 7020 can also evaluate resource usage or other operating configurations of various medical facilities to determine whether resource usage can be improved. Similarly, the recommendation module 7030 can be configured to analyze the aggregated organizational data from the data collection and aggregation module 7022 to provide recommendations. For example, the recommendation module 7030 can recommend to a medical facility (e.g., a health care provider such as a hospital) that a particular surgical instrument 7012 should be upgraded to an improved version based, for example, on a higher than expected error rate. Additionally, the recommendation module 7030 and / or resource optimization module 7020 can recommend better supply chain parameters, such as product reorder points, and provide suggestions for different surgical instruments 7012, their use, or procedure steps that will improve surgical outcomes. The medical facility can receive such recommendations via the corresponding surgical hub 7006. More specific recommendations regarding the parameters or configurations of various surgical instruments 7012 can also be provided. The hub 7006 and / or surgical instruments 7012 can also each have a display screen that displays the data or recommendations provided by the cloud 7004.

[0153] The patient outcome analysis module 7028 may analyze surgical outcomes associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 may also analyze and evaluate other potential operating parameters. In this regard, the recommendation module 7030 may use these other potential operating parameters to make recommendations based on resulting in better surgical outcomes, such as a better seal or less bleeding. For example, the recommendation module 7030 may be able to send suggestions to the surgical instrument 7006 regarding when to use a particular cartridge with a corresponding stapling surgical instrument 7012. Thus, the cloud-based analysis system may be configured to analyze large-scale collected raw data and provide centralized recommendations (advantageously determined based on aggregated data) across multiple medical facilities while controlling for common variables. For example, the cloud-based analysis system may analyze, evaluate, and / or aggregate type of medical procedure, type of patient, number of patients, geographic similarities between medical providers using similar types of instruments, etc., in ways that no single medical facility could analyze independently on its own. The control program update module 7026 can be configured to implement recommendations for various surgical instruments 7012 when the corresponding control programs are updated. For example, the patient outcome analysis module 7028 can identify correlations linking particular control parameters to successful (or unsuccessful) outcomes. Such correlations can be addressed when an updated control program is transmitted to the surgical instrument 7012 via the control program update module 7026. Updates to the surgical instrument 7012, which can be transmitted via the corresponding hub 7006, may incorporate aggregated performance data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. Additionally, the patient outcome analysis module 7028 and recommendation module 7030 can identify improved methods of using the surgical instrument 7012 based on the aggregated performance data.

[0154] The cloud-based analysis system may include security features implemented by the cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other suitable security credentials. These credentials may be stored in memory 7010 and associated with an authorized cloud access level. For example, based on providing accurate credentials, the surgical hub 7006 may be granted access to communicate with the cloud to a predetermined extent (e.g., send or receive certain defined types of information). To this end, the cloud 7004's aggregated medical data database 7011 may include a database of certified credentials to verify the accuracy of the provided credentials. Different credentials may be associated with various levels of permission for interaction with the cloud 7004, such as a predetermined access level for receiving data analyses generated by the cloud 7004. Furthermore, for security purposes, the cloud may maintain a database of hubs 7006, surgical instruments 7012, and other devices, which may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be permitted to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be prevented from fully functioning when paired with the corresponding hub 7006. Additionally or alternatively, the cloud 7004 may flag surgical instruments 7012 based on incompatibility or other specified criteria. In this manner, counterfeit medical devices and the inappropriate reuse of such devices across the cloud-based analysis system may be identified and addressed.

[0155] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal that may represent, for example, authorization credentials for access to the corresponding hub 7006 and the cloud 7004. A wired transceiver may also be used to transmit the signal. Such authorization credentials may be stored in a respective memory device of the surgical instrument 7012. The authorization and security module 7024 may determine whether the authorization credentials are accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credentials may also be encrypted, such as by using hash-based encryption. Upon transmitting the appropriate authorization, the surgical instrument 7012 may transmit a signal to the corresponding hub 7006 and ultimately the cloud 7004 indicating that the surgical instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may transition to a state capable of receiving medical data for storage in the aggregated medical data database 7011. This readiness to transmit data may be indicated, for example, by a light indicator on the surgical instrument 7012. The cloud 7004 may also send signals to the surgical instruments 7012 to update their associated control programs. The cloud 7004 may send signals directed to a particular class of surgical instruments 7012 (e.g., electrosurgical instruments) so that software updates to control programs are sent only to the appropriate surgical instruments 7012. Additionally, the cloud 7004 may be used to implement system-wide solutions to address local or global issues based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, the cloud 7004 may change the authorization credentials corresponding to this group to implement an operational lockout for this group.

[0156] The cloud-based analytics system may enable monitoring of multiple healthcare facilities (e.g., healthcare facilities such as hospitals) to determine improved practices and recommend changes accordingly (e.g., via recommendation module 2030). Thus, the processor 7008 of the cloud 7004 may analyze data associated with an individual healthcare facility to identify the facility and aggregate that data with other data associated with other healthcare facilities. Groups may be defined, for example, based on similar operational behavior or geographic location. In this manner, the cloud 7004 may provide broader analysis and recommendations for groups of healthcare facilities. The cloud-based analytics system may also be used for enhanced situational awareness. For example, the processor 7008 may predictively model the effect of recommendations on cost and effectiveness for a particular facility (compared to overall operations and / or various healthcare procedures). The costs and effectiveness associated with that particular facility may also be compared to the corresponding local area of ​​other facilities or any other comparable facilities.

[0157] The data classification and prioritization module 7032 may prioritize and classify data based on criticality (e.g., the severity, surprise, or suspiciousness of the medical event associated with the data). This classification and prioritization may be used in conjunction with other data analysis module 7034 functionality described herein to improve the cloud-based analyses and operations described herein. For example, the data classification and prioritization module 7032 may assign priorities to data analyses performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels may result in specific responses from the cloud 7004 (corresponding to the level of urgency), such as elevation for rapid response, special handling, exclusion from the aggregated medical data database 7011, or other suitable responses. Additionally, if necessary, the cloud 7004 may send a request (e.g., a push message) via the hub application server for additional data from the corresponding surgical instrument 7012. The push message may result in a notification being displayed on the corresponding hub 7006 to request support or additional data. This push message may be needed in situations where the cloud detects a significant irregularity or outlier and the cloud is unable to determine the cause of the irregularity. The central server 7013 can be programmed to trigger this push message in certain critical situations, such as when data is determined to differ from expected values ​​by more than a predetermined threshold, or when security is suspected.

[0158] Further exemplary details regarding the various described functions are provided in the following description, each of which may utilize a cloud architecture, as illustrated in Figures 11 and 12 as an example of a hardware and software implementation.

[0159] 13 shows a block diagram of a computer-implemented adaptive surgical system 9060 configured to adaptively generate control program updates for modular devices 9050, in accordance with at least one embodiment of the present disclosure. In some examples, the surgical system may include a surgical hub 9000, a plurality of modular devices 9050 communicatively coupled to the surgical hub 9000, and an analysis system 9100 communicatively coupled to the surgical hub 9000. While a single surgical hub 9000 is shown, it should be noted that the surgical system 9060 may include any number of surgical hubs 9000 that may be connected to form a network of surgical hubs 9000 communicatively coupled to the analysis system 9100. In some examples, the surgical hub 9000 may include a processor 9010 coupled to a memory 9020 for executing instructions stored thereon and a data relay interface 9030 through which data is transmitted to the analysis system 9100. In some examples, the surgical hub 9000 may further include a user interface 9090 having an input device 9092 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 9094 (e.g., a display screen) for providing output to the user. The output may include data from a query entered by the user, suggestions for products or product mixtures to use in a given procedure, and / or instructions for actions to be performed before, during, or after a surgical procedure. The surgical hub 9000 may further include an interface 9040 for communicatively coupling a modular device 9050 to the surgical hub 9000. In one aspect, the interface 9040 may include a transceiver communicatively connectable to the modular device 9050 via a wireless communication protocol. The modular device 9050 may include, for example, a surgical stapling and severing instrument, an electrosurgical instrument, an ultrasonic instrument, an injector, a respirator, and a display screen. In some instances, the surgical hub 9000 may further be communicatively coupled to one or more patient monitoring devices 9052, such as an EKG monitor or a BP monitor.In some examples, the surgical hub 9000 may further be communicatively coupled to one or more databases 9054 or external computer systems, such as an EMR database of the medical facility in which the surgical hub 9000 is located.

[0160] When a modular device 9050 is connected to the surgical hub 9000, the surgical hub 9000 can sense or receive pre- and post-operative data from the modular device 9050 and then associate the received pre- and post-operative data with surgical procedure outcome data. The pre- and post-operative data can indicate how the modular device 9050 was controlled during the course of a surgical procedure. The procedure outcome data includes data associated with the results from the surgical procedure (or steps thereof), which can include whether the surgical procedure (or steps thereof) had a positive or negative outcome. For example, the outcome data can include whether a patient suffered a post-operative complication from a particular procedure or whether there was a leak (e.g., bleeding or air leak) at a particular staple or incision line. The surgical hub 9000 can obtain the surgical procedure outcome data by receiving data from an external source (e.g., from the EMR database 9054), by directly detecting the outcome (e.g., via one of the connected modular devices 9050), or by inferring the occurrence of the outcome through a situational awareness system. For example, data regarding post-operative complications can be retrieved from the EMR database 9054, and data regarding staple or incision line leakage can be directly detected or inferred by the situational awareness system. Surgical procedure outcome data can be inferred by the situational awareness system from data received from a variety of data sources, including the modular device 9050 itself, the patient monitoring device 9052, and the database 9054 to which the surgical hub 9000 is connected.

[0161] The surgical hub 9000 can transmit associated modular device 9050 data and outcome data to the analysis system 9100 for processing thereon. By transmitting both pre- and post-operative data indicating how the modular device 9050 is controlled and procedure outcome data, the analysis system 9100 can correlate different methods of controlling the modular device 9050 with surgical outcomes for specific procedure types. In some examples, the analysis system 9100 may include a network of analysis servers 9070 configured to receive data from the surgical hub 9000. Each of the analysis servers 9070 may include a memory and a processor coupled to the memory that executes instructions stored thereon to analyze the received data. In some examples, the analysis servers 9070 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 9100 can then learn optimal or preferred operating parameters for various types of modular devices 9050, generate adjustments to the control programs of the modular devices 9050 in the field, and then send (or "push") the updates to the control programs of the modular devices 9050.

[0162] Additional details regarding the computer-implemented interactive surgical system 9060, including the surgical hub 9000 and various modular devices 9050 connectable thereto, are described in connection with Figures 5-6.

[0163] 14 provides a surgical system 6500 according to the present disclosure and may include a surgical instrument 6502 that can communicate with a console 6522 or a portable device 6526 through a local area network 6518 or a cloud network 6520 via a wired or wireless connection. In various aspects, the console 6522 and the portable device 6526 may be any suitable computing device. The surgical instrument 6502 may include a handle 6504, an adapter 6508, and a loading unit 6514. The adapter 6508 releasably couples to the handle 6504, and the loading unit 6514 releasably couples to the adapter 6508 such that the adapter 6508 transfers force from the drive shaft to the loading unit 6514. The adapter 6508 or the loading unit 6514 may include a force gauge (not expressly shown) disposed therein to measure force exerted on the loading unit 6514. The loading unit 6514 can include an end effector 6530 having a first jaw 6532 and a second jaw 6534. The loading unit 6514 can be a field loading or multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without the loading unit 6514 needing to be removed from the surgical site to reload the loading unit 6514.

[0164] The first and second jaws 6532, 6534 may be configured to clamp tissue therebetween, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 6532 may be configured to fire at least one fastener multiple times or may be configured to contain a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that may be fired two or more times before being replaced. The second jaw 6534 may include an anvil that deforms or otherwise secures fasteners around tissue as they are ejected from the multi-fire fastener cartridge.

[0165] The handle 6504 can include a motor coupled to the drive shaft to affect rotation of the drive shaft. The handle 6504 can include a control interface for selectively activating the motor. The control interface can include buttons, switches, levers, sliders, a touch screen, and any other suitable input mechanism or user interface that can be engaged by a clinician to activate the motor.

[0166] The control interface of the handle 6504 may communicate with a controller 6528 of the handle 6504 to selectively activate the motors to affect rotation of the drive shaft. The controller 6528 may be disposed within the handle 6504 and configured to receive input from the control interface and adapter data from the adapter 6508 or loading unit data from the loading unit 6514. The controller 6528 may analyze the input from the control interface and the data received from the adapter 6508 and / or the loading unit 6514 to selectively activate the motors. The handle 6504 may also include a display viewable by a clinician while using the handle 6504. The display may be configured to display portions of the adapter or loading unit data before, during, or after firing of the instrument 6502.

[0167] The adapter 6508 may include an adapter identification device 6510 disposed therein, and the loading unit 6514 includes a loading unit identification device 6516 disposed therein. The adapter identification device 6510 may be in communication with a controller 6528, and the loading unit identification device 6516 may be in communication with the controller 6528. It will be understood that the loading unit identification device 6516 may be in communication with the adapter identification device 6510, which relays or passes through communications from the loading unit identification device 6516 to the controller 6528.

[0168] The adapter 6508 may also include several sensors 6512 (one shown) disposed about its periphery to detect various conditions of the adapter 6508 or the environment (e.g., when the adapter 6508 is connected to the loading unit, when the adapter 6508 is connected to the handle, when the drive shaft is rotating, the torque of the drive shaft, the strain on the drive shaft, the temperature within the adapter 6508, the number of firings of the adapter 6508, the peak force of the adapter 6508 during firing, the total amount of force applied to the adapter 6508, the peak retraction force of the adapter 6508, the number of times the adapter 6508 is idle during firing, etc.). The multiple sensors 6512 can provide input to the adapter identification device 6510 in the form of data signals. The data signals of the multiple sensors 6512 may be stored in the adapter identification device 6510 or may be used to update the adapter data stored in the adapter identification device 6510. The data signals of the multiple sensors 6512 may be analog or digital. The plurality of sensors 6512 may include a force gauge for measuring the force exerted on the loading unit 6514 during firing.

[0169] The handle 6504 and adapter 6508 may be configured to interconnect the adapter identification device 6510 and the loading unit identification device 6516 with the controller 6528 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage each other and transmit energy and signals therebetween). Additionally or alternatively, the electrical interface may be a contactless electrical interface for wirelessly transmitting energy and signals therebetween (e.g., by inductive transfer). It is also contemplated that the adapter identification device 6510 and the controller 6528 may wirelessly communicate with each other via a wireless connection that is separate from the electrical interface.

[0170] The handle 6504 may include a transmitter 6506 configured to transmit instrument data from the controller 6528 to other components of the system 6500 (e.g., the LAN 6518, the cloud 6520, the console 6522, or the portable device 6526). The transmitter 6506 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 6500. For example, the controller 6528 may transmit instrument data to the console 6522, including the serial number of an attached adapter (e.g., adapter 6508), the serial number of a loading unit (e.g., loading unit 6514) attached to the adapter, and the serial number of a multi-fire fastener cartridge (e.g., multi-fire fastener cartridge) loaded in the loading unit attached to the handle 6504. The console 6522 may then transmit data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, back to the controller 6528. The controller 6528 can display a message on the local instrument display or send a message via transmitter 6506 to the console 6522 or portable device 6526, which can display the message on the display 6524 or portable device screen, respectively.

[0171] 15A shows an exemplary flow for determining an operating mode and operating in the determined mode. The computer-implemented interactive surgical system and / or its components and / or subsystems may be configured to be updated. Such updates may include the inclusion of features and benefits that were not available to the user prior to the update. These updates may be established by any method of hardware, firmware, and software update suitable for introducing functionality to the user. For example, interchangeable / swappable (e.g., hot-swappable) hardware components, flashable firmware devices, and updatable software systems may be used to update the computer-implemented interactive surgical system and / or its components and / or subsystems.

[0172] An update may be contingent on any suitable criterion or set of criteria. For example, an update may be contingent on one or more hardware capabilities of the system, such as processing power, bandwidth, resolution, etc. For example, an update may be contingent on one or more software aspects, such as the purchase of specific software code. For example, an update may be contingent on a purchased service tier. A service tier may represent a feature and / or set of features that a user is entitled to use in connection with the computer-implemented interactive surgical system. A service tier may be determined by a license code, an e-commerce server authentication interaction, a hardware key, a username / password combination, a biometric authentication interaction, a public / private key exchange interaction, etc.

[0173] At 10704, a system / device parameter may be identified. A system / device parameter may be any element or set of elements upon which an update is conditioned. For example, the computer-implemented interactive surgical system may detect a certain bandwidth of communication between a modular device and a surgical hub. For example, the computer-implemented interactive surgical system may detect an indication to purchase a certain service tier.

[0174] At 10708, an operational mode may be determined based on the identified system / device parameters. This determination may be made by a process that maps system / device parameters to operational modes. The process may be a manual and / or automatic process. The process may be the result of local and / or remote computation. For example, a client / server interaction may be used to determine the operational mode based on the identified system / device parameters. For example, local software and / or locally embedded firmware may be used to determine the operational mode based on the identified system / device parameters. For example, a hardware key, such as a secure microprocessor, may be used to determine the operational mode based on the identified system / device parameters.

[0175] At 10710, operation may proceed according to the determined operating mode. For example, the system or device may proceed to operate in a default operating mode. For example, the system or device may proceed to operate in an alternate operating mode. The operating mode may be dictated by control hardware, firmware, and / or software already resident within the system or device. The operating mode may be dictated by newly installed / updated control hardware, firmware, and / or software.

[0176] FIG. 15B shows an example functional block diagram for changing the operational mode. The upgradeable element 10714 may include an initialization component 10716. The initialization component 10716 may include any hardware, firmware, and / or software suitable for determining the operational mode. For example, the initialization component 10716 may be part of a system or device startup procedure. The initialization component 10716 may be involved in an interaction to determine the operational mode of the upgradeable element 10714. For example, the initialization component 10716 may interact with, for example, a user 10730, an external resource 10732, and / or a local resource 10718. For example, the initialization component 10716 may receive a license key from the user 10730 to determine the operational mode. The initialization component 10716 may query an external resource 10732, such as a server, using the serial number of the upgradeable element 10714 to determine the operational mode. For example, the initialization component 10716 may query local resources 10718, such as a local query to determine the amount of available bandwidth and / or a local query of a hardware key to determine the operating mode, for example.

[0177] The upgradeable element 10714 may include one or more operational components 10720, 10722, 10726, 10728 and an operational pointer 10724. The initialization component 10716 can instruct the operational pointer 10724 to direct operation of the upgradeable element 10741 to the operational components 10720, 10722, 10726, 10728 that correspond to the determined operational mode. The initialization component 10716 can instruct the operational pointer 10724 to direct operation of the upgradeable element to the default operational component 10720. For example, the default operational component 10720 may be selected on the condition that no other alternative operational mode has been determined. For example, the default operational component 10720 may be selected on the condition that the initialization component fails and / or an interaction fails. The initialization component 10716 can instruct the operational pointer 10724 to instruct the resident operational components 10722 to operate the upgradeable components 10714. For example, certain functionality may reside in the upgradeable components 10714 but require activation to operate. The initialization component 10716 can instruct the operational pointer 10724 to instruct the operation of the upgradeable components 10714 to install new operational components 10728 and / or newly installed operational components 10726. For example, new software and / or firmware may be downloaded. The new software and / or firmware may include code to enable features represented by a selected operational mode. For example, new hardware components may be installed to enable the selected operational mode.

[0178] Coordination between the primary and / or secondary displays may be provided, for example, coordination between the local fixture display and the paired imaging device display.

[0179] Instruments may be provided that may include a local display and a hub with an operating room (OR) or operating room display separate from the instrument display. When an instrument is coupled to a surgical hub, the secondary display on the device reconfigures to display different information than when it is independent of the surgical hub connection. Some of the information on the instrument's secondary display may be displayed on the surgical hub's primary display. Image fusion may be performed, which may allow for the overlay of one or more of the following: instrument status, integrated landmarks used to interlock several images, and guidance features. Image fusion may be provided on the surgical hub and / or instrument display. As disclosed herein, several techniques for overlaying or augmenting images and / or text from multiple image / text sources may be used to present a composite image on one or more displays.

[0180] Collaboration between one or more local instrument displays and a paired laparoscope display may be provided. The behavior of the instrument's local display may change when it senses the connectable presence of a display (e.g., a global display) that may be coupled to the surgical hub. The present disclosure may provide a 360￢∞ composite top view of the surgical site, which may assist in avoiding collateral structures.

[0181] During a surgical procedure, the surgical site may be displayed on a remote surgical hub display. The remote surgical hub display may be referred to as the primary display. During a surgical procedure, the surgical device may track and record surgical data and variables (e.g., surgical parameters) that may be stored within the instrument (see Figures 1-13 for instrument architecture including processor, memory, control circuitry, storage, etc.). Surgical parameters may include firing force (FTF), force-to-close (FTC), firing progression, tissue gap, power level, impedance, tissue compressive stability (creep), etc. Providing image / text overlays may be provided, for example, to allow the surgeon to view a display that may present overlaid image / text information.

[0182] When a surgical device (e.g., an instrument) is connected to the surgical hub, a composite image may be displayed on the primary display that may include a view of the surgical site received from a first instrument (e.g., a medical imaging device such as a laparoscope, endoscope, thoracoscope, etc.), which may be augmented with surgical data and variables received from a second instrument (e.g., a surgical stapler) to provide related images and data on the primary display.

[0183] During a surgical procedure, the surgical site may be displayed on the primary surgical hub display as a narrow field of view of the medical imaging device. Items outside the current field of view, i.e., collateral structures, cannot be seen without moving the medical imaging device.

[0184] One embodiment can provide a narrow view of the surgical site in a first window of the display augmented by a wider view of the surgical site in a separate window of the display, providing a combined overhead view that is mapped using two or more imaging arrays to provide an augmented image of multiple perspectives of the surgical site.

[0185] One embodiment may provide a wide view of the surgical site on a first display, which may be a primary display, and a narrower view of the surgical site on a second display, which may be a secondary display.

[0186] A surgical hub may be provided that may include a processor and a memory coupled to the processor. The memory may store instructions executable by the processor to detect a surgical device connection to the surgical hub, transmit control signals to the detected surgical device to transmit surgical parameter data associated with the detected device to the surgical hub, receive the surgical parameter data, receive image data from the image sensor, and display, on a display coupled to the surgical hub, the image received from the image sensor along with the surgical parameter data received from the surgical device.

[0187] In another aspect, the present disclosure provides a surgical hub including a processor and a memory coupled to the processor. The memory can store instructions executable by the processor to receive first image data from a first image sensor, receive second image data from a second image sensor, and display a first image corresponding to the first field of view and a second image corresponding to the second field of view on a display coupled to the surgical hub. The first image data represents the first field of view, and the second image data represents the second field of view. The display can be a primary display and / or a secondary display. The display can be inside the sterile field or outside the sterile field.

[0188] The first field of view may be a narrow-angle field of view, and the second field of view may be a wide-angle field of view. The first image may be augmented with the second image on the display. The first image may be fused with the second image into a third image, and the fused image may be displayed on the display. The fused image data may include instrument data, which may include status information associated with the surgical device, image data integration landmarks for interlocking multiple images, guidance parameters, etc. The first image sensor may capture the first image data at a first time and the second image data at a second time.

[0189] Third image data can be received from a third image sensor, and the third image data can represent a third field of view. Composite image data can be generated that includes the second and third image data. The first image can be displayed on a first display and / or in a first window on the display. The first image can correspond to the first image data. The third image can be displayed on a second display and / or in a second window on the first display. The third image can correspond to the composite image data. The display can be a primary display and / or a secondary display. The display can be inside the sterile field or outside the sterile field.

[0190] The third image data can represent a third field of view. The second image data can be fused with the third image data to generate fused image data. The first image can be displayed on a first display and / or in a first window on the display. The first image can correspond to the first image data. The third image can be displayed on a second display and / or in a second window on the first display. The third image can correspond to the fused image data.

[0191] Displaying an endoscopic image augmented with a surgical device image on the primary surgical hub display may allow a surgeon to focus on the display and obtain a view of the surgical site augmented with surgical device data associated with the surgical procedure, such as firing force, closure force, firing progression, tissue gap, power level, impedance, tissue compressive stability (creep), etc. The endoscopic image may be augmented with the surgical device image and displayed on the primary display and / or secondary display. For example, the primary display may display the endoscopic image augmented with the surgical device image, and the secondary display may display the surgical device image. As described herein, a user may gesture and / or issue commands to change the primary and / or secondary display. For example, a user can move an image displayed on a secondary display onto the primary display, or vice versa. Displaying a narrow field of view image in a first window on the display and a composite image of some other perspective, such as a wider field of view, allows a surgeon to simultaneously view a magnified image of the surgical site with a wider field of view of the surgical site without moving the scope.

[0192] Both global and local displays of devices (e.g., surgical instruments) may be provided. The local display may be coupled to the surgical hub. The global display may be associated with a primary display. The local display may be associated with a secondary display. The device may display one or more (e.g., all) of its associated menus on the local display until it senses a connection to the surgical hub, at which point a subset of the information may be displayed on the primary display, e.g., a monitor, through the surgical hub. Information may or may not be mirrored on the device display. Information may be removed from the device screen. This technique frees up the device display to show different information or display larger font information on the surgical hub display.

[0193] The instrument may have a local display, which may be a secondary display. The surgical hub may be associated with an operating room (e.g., operating room or OR) display, which may be separate from the instrument display and may be the primary display. When an instrument is coupled to the surgical hub, the instrument local display may become a secondary display and may be reconfigured to display different information than when the instrument may operate independently of the surgical hub connection. In another aspect, some of the information on the secondary display may be displayed on the primary display in the operating room through the surgical hub.

[0194] FIG. 16 illustrates a primary display of a surgical hub. For example, FIG. 16 illustrates an exemplary primary display 6200 associated with a surgical hub 206 comprising a global display window 6202 and a local instrument display window 6204, according to one aspect of the present disclosure. With continued reference to FIGS. 1-11 illustrating interaction with the interactive surgical system 100 environment, including the surgical hubs 106, 206, and FIGS. 12-14, which together illustrate instruments connected to the surgical hub, the behavior of the local instrument display 6204 can be displayed when an instrument 235 senses the connectable presence of the global display window 6202 through the surgical hub 206. The global display window 6202 may, for example, in the center of the surgical hub display 215, also referred to herein as a monitor, show a view 6206 of a surgical site 6208 as seen through a medical imaging device, such as a laparoscope / endoscope 219 coupled to an imaging module 238. The end effector 6218 portion of the connected instrument 235 may be shown within a field of view 6206 of the surgical site 6208 in the global display window 6202. The image shown on the display 237 located on the instrument 235 coupled to the surgical hub 206 is shown or mirrored on the local instrument display window 6204 located in the lower right corner of the monitor 6200, for example, as shown in FIG.

[0195] In operation, associated instruments and information and menus may be displayed on the display 237 located on the instrument 235 until the instrument 235 senses connection of the instrument 235 to the surgical hub 206, at which point all or some subset of the information presented on the instrument display 237 may be displayed (e.g., only) on the local instrument display window 6204 portion of the surgical hub display 6200 through the surgical hub 206. The information displayed on the local instrument display window 6204 may be mirrored on the display 237 located on the instrument 235 or may no longer be accessible on the start-up screen of the instrument display 237. This technique frees the instrument 235 to show different information or larger font information on the surgical hub display 6200.

[0196] The primary display 6200 may provide pre- and post-operative visualization of the surgical site 6208. Advanced imaging may identify and visually highlight (6222) critical structures such as the ureter 6220 (or nerves, etc.), which can be tracked and shown on the left side of the display 6200 by the instrument proximity display 6210. In the illustrated example, the instrument proximity display 6210 can show instrument-specific settings. For example, the upper instrument proximity display 6212 can show settings for monopolar instruments, the middle instrument proximity display 6214 can show settings for bipolar instruments, and the lower instrument proximity display 6212 can show settings for ultrasound instruments.

[0197] One or more secondary displays, which may be dedicated local displays, may be linked to the surgical hub 206 to provide both an interaction portal via a touchscreen display and / or a secondary screen that may display any number of the surgical hub's 206's tracked data feeds to provide status. The secondary screen may display firing force (FTF), tissue gap, power level, impedance, tissue compressive stability (creep), etc., while the primary screen may display key variables (e.g., only key variables) to keep the feed clutter-free. The interactive display may be used to move the display of information on the primary display to a desired position, size, color, etc. For example, a user may use the interactive display to move information onto the primary display, where it may be highlighted and / or displayed more prominently than other data.

[0198] 16, the secondary screen displays an instrument proximity display 6210 on the left side of the display 6200 and a local instrument display 6204 on the lower right side of the display 6200. The local instrument display 6204 presented on the surgical hub display 6200 displays an icon of the end effector 6218, for example, an icon of the staple cartridge 6224 currently in use, the size 6226 of the staple cartridge 6224 (e.g., 60 mm), and the current position of the knife 6228 of the end effector.

[0199] A display 237 located on the instrument 235 can display the wireless or wired attachment of the instrument 235 to the surgical hub 206 and the instrument's communications / logging on the surgical hub 206. Settings may be provided on the instrument 235 to allow the user to select mirroring or extending the display on both monitoring devices. The instrument controls may be used to interact with the surgical hub display of information being provided on the instrument. As disclosed herein, the instrument 235 may include wireless communication circuitry for communicating wirelessly with the surgical hub 206.

[0200] A first instrument coupled to the surgical hub 206 can be paired with the screen of a second instrument coupled to the surgical hub 206, allowing both instruments to display some hybrid combination of information from both two devices, mirroring a portion of the primary display.

[0201] The primary display 6200 of the surgical hub 206 can provide a 360° composite top view of the surgical site 6208 to avoid collateral structures. For example, a secondary display of an end effector surgical stapler may be provided within the primary display 6200 of the surgical hub 206 or on a separate display to provide a better view around the area within the current field of view 6206.

[0202] FIG. 17 shows an example of a primary display of a surgical hub. For example, FIG. 17 can show an exemplary primary display having a composite overhead view of a portion of a surgical stapler end effector 6234 mapped using two or more imaging arrays or one array, with time providing multiple perspective views of the end effector 6234 to enable composite imaging of the overhead view. The techniques described herein may be applied to ultrasonic instruments, electrosurgical instruments, combination ultrasonic / electrosurgical instruments, and / or combination surgical stapler / electrosurgical instruments. Several techniques may be implemented to overlay or augment images and / or text from multiple image / text sources to present a composite image on a display (e.g., a single display).

[0203] As shown in FIG. 17 , the primary display 6200 of the surgical hub 206 can display a primary window 6230. The primary window 6230 may be located in the center of the screen and shows a magnified or exploded narrow-angle view of the surgical field of view 6232. The primary window 6230, located in the center of the screen, shows a magnified or narrow-angle view of the end effector 6234 of a surgical stapler gripping a blood vessel 6236. The primary window 6230 can display a tissue image to generate a composite image allowing visualization of structures adjacent to the surgical field of view 6232. A second window 6240 may be shown in the lower left corner of the primary display 6200. The second window 6240 displays the tissue image in a wide-angle view with a standard focus of the image shown in the primary window 6230 in an overhead view. The overhead view provided in the second window 6240 may allow the observer to easily view items outside the narrow surgical field of view 6232 without moving the laparoscope or other imaging device coupled to the imaging module 238 of the surgical hub 206. The third window 6242 may be displayed in the lower right corner of the primary display 6200 and shows an icon 6244 representing the staple cartridge (e.g., staple cartridge in this example) of the end effector 6234 and additional information such as "4 rows" indicating the number of staple rows 6246 and "35 mm" indicating the distance 6248 traversed by the knife along the length of the staple cartridge. Below the third window 6242 is displayed an icon 6258 of a frame of the current state of the clamp stabilization sequence 6250 indicating clamp stabilization.

[0204] In an exemplary visualization control mode, the display may be controlled by the user, for example, via motion tracking (e.g., head orientation relative to the monitor), hand gestures, voice activation, and other means within the sterile field. The user may use gestures, motion tracking commands, voice activation, etc. to move data from one display to another. For example, the user may use gestures to move data from a first display to a second display. The gestures may be detected by the hub, which may instruct the first display to delete or stop displaying the data and instruct the second display to display the data.

[0205] 18 shows a diagram of four wide-angle view images of a surgical site at four separate times during a procedure. For example, FIG. 18 shows a diagram 6270 of four separate wide-angle view images 6272, 6274, 6276, 6278 of a surgical site at four separate times during a procedure, according to one embodiment of the present disclosure.

[0206] The sequence of images illustrates the creation of overhead composite images in wide and narrow focus over time. A first image 6272 is a wide-angle view of the end effector 6234 clamping the vessel 6236 taken at a prior time t0 (e.g., 09:35:09). A second image 6274 is another wide-angle view of the end effector 6234 clamping the vessel 6236 taken at a current time t1 (e.g., 09:35:13). A third image 6276 is a composite image of an overhead view of the end effector 6234 clamping the vessel 6236 taken at the current time t1. The third image 6276 may be displayed in a second window 6240 of the primary display 6200 of the surgical hub 206, as shown in FIG. 17 . The fourth image 6278 is a narrow angle view of the end effector 6234 clamping the blood vessel 6236 at the current time t1 (e.g., 09:35:13). The fourth image 6278 is a narrow angle view of the surgical site shown in the primary window 6230 of the primary display 6200 of the surgical hub 206, as shown in FIG.

[0207] In one aspect of the present disclosure, the primary display and / or the secondary display may display one or more of a first image, a second image, a third image, and / or a fourth image. For example, the primary display may display the third image and the secondary display may display the fourth image. As another example, the primary display may display the fourth image and the secondary display may display the third image.

[0208] 19 shows an example of an augmented video image of a preoperative video image augmented with data identifying the displayed element. The preoperative video image, which may be augmented with data, may be displayed on a primary display and / or a secondary display. For example, the augmented video image may be displayed on the primary display and the video image may be displayed on the secondary display. As another example, the augmented video image may be displayed on the secondary display, while the video image may be displayed on the primary display.

[0209] 19 shows an example of an augmented video image 6350 including a pre-operative video image 6352 augmented with data (e.g., 6354, 6356, 6358 identifying displayed elements). An augmented reality vision system may be employed in a surgical procedure to implement a method for augmenting data onto the pre-operative image 6352. The method includes generating a pre-operative image 6352 of an anatomical division of a patient and generating an augmented video image of a surgical site within the patient. The augmented video image 6350 may include an image of at least a portion of a surgical tool 6354 manipulated by a user 6456. The method may further include processing the pre-operative image 6352 to generate data related to the anatomical division of the patient. The data may include a label 6358 of the anatomical division and a peripheral margin of at least a portion of the anatomical division. The peripheral margin can be configured to guide the surgeon to a cutting position relative to the anatomical section, embed data and a user's 6356 identification information within the pre-operative image 6350, and display an augmented video image 6350 of the patient's anatomical section to the user. The method may further include sensing a load condition on the surgical tool 6354, generating a feedback signal based on the sensed load condition, and updating in real time the data and location of the identification information of the user operating the surgical tool 6354 embedded within the augmented video image 6350 in response to a change in the location of the surgical tool 6354 within the augmented video image 6350. A further example is disclosed in U.S. Patent No. 9,123,155, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES," issued September 1, 2015, which is incorporated herein by reference in its entirety.

[0210] In one aspect, radiography integration technology can be used to overlay the pre-operative images 6352 with data acquired through live internal sensing or pre-procedure technology. Radiography integration may include surgical landmarks, marker and landmark identification using radiographic markers placed inside or outside the patient, and identification of radiopaque staples, clips, or other tissue fixation articles. Digital radiography technology can be used to generate digital images for overlay with the pre-operative images 6352. Digital radiography is a form of x-ray imaging that uses a digital image capture device with a digital x-ray sensor instead of traditional photographic film. Digital radiography technology provides instant image preview and availability for overlay with the pre-operative images 6352. Additionally, special image processing techniques can be applied to the digital x-ray images to improve the overall display quality of the images.

[0211] Digital radiography techniques may employ image detectors, including flat-panel detectors (FPDs), which may be divided into two categories: indirect FPDs and direct FPDs. Indirect FPDs may include amorphous silicon (a-Si) made from cesium iodide (CSI) or gadolinium oxysulfide (Gd2O2S) combined with a scintillator in the outer layer of the detector to convert X-rays into light. The light may be guided through an a-Si photodiode layer, where it is converted into a digital output signal. The digital signal may then be read out by a thin-film transistor (TFT) or fiber-coupled charge-coupled device (COD). Direct FPDs include amorphous selenium (a-Se) FPDs, which directly convert X-ray photons into electrical charges. The outer layer of the flat panel in this design may be a high-voltage bias electrode. X-ray photons may generate electron-hole pairs in the a-Se, and the passage of these electrons and holes may depend on the potential of the bias voltage charge. As holes can be replaced with electrons, the resulting charge pattern in the selenium layer can be read out by TFT arrays, active matrix arrays, electrometer probes, or microplasma line addressing. Other direct digital detectors may be based on CMOS and CCD technology. Phosphor detectors may also be used to record the X-ray energy during exposure, and may be scanned by a laser diode to excite the stored energy, which may be released and read out by a CCD digital image capture array.

[0212] FIG. 20 illustrates an exemplary flow diagram of a process for displaying one or more images. For example, FIG. 20 illustrates a logic flow diagram 6360 of a process illustrating a control program or logical configuration for displaying images, according to one embodiment of the present disclosure. Referring also to FIGS. 1-11 to illustrate interaction with an interactive surgical system 100 environment including a surgical hub 106, 206, the present disclosure, in one embodiment, provides a surgical hub 206 comprising a processor 244 and a memory 249 coupled to the processor 244. The memory 249 stores instructions executable by the processor 244 to receive (6362) first image data from a first image sensor, receive (6364) second image data from a second image sensor, and display (6366) a first image corresponding to the first field of view and a second image corresponding to the second field of view on a display. The first image data can represent the first field of view, and the second image data can represent the second field of view. The display can be a primary display and / or a secondary display. The display may be a display 217 coupled to the surgical hub 206 .

[0213] In one aspect, the first field of view can be a narrow field of view and the second field of view can be a wide field of view. In another aspect, the memory 249 stores instructions executable by the processor 244 to augment the first image with the second image on the display. The display can be a primary display and / or a secondary display.

[0214] In another aspect, memory 249 stores instructions executable by processor 244 to fuse the first image and the second image into a third image and display the fused image on a display. The display may be a primary display and / or a secondary display. The display may be display 217. The first image, the second image, and / or the third image may be displayed on the secondary display, while the fused image may be displayed on the primary display. The first image, the second image, and / or the third image may be displayed on the primary display, while the fused image may be displayed on the secondary display.

[0215] In another aspect, the fused image data includes status information associated with the surgical device 235, image data integration landmarks for interlocking the images, and at least one guidance parameter. In another aspect, the first image sensor is the same image sensor, and the first image data is captured at a first time and the second image data is captured at a second time. The one or more images can be displayed on the primary display and / or the secondary display.

[0216] In another aspect, memory 249 stores instructions executable by processor 244 to receive third image data from a third image sensor (the third image data representing a third field of view), generate composite image data including the second and third image data, and display the first image in a first window on the display (the first image corresponds to the first image data) and display the third image in a second window on display 215 (the third image corresponds to the composite image data). In another aspect, the first image, the second image, and / or the third image may be displayed on the primary display and / or the secondary display. For example, a user may indicate that the primary display and / or the secondary display may display one of the first image, the second image, and the third image.

[0217] In another aspect, memory 249 stores instructions executable by processor 244 to receive third image data from a third image sensor (the third image data representing a third field of view), fuse the second image data and the third image data to generate fused image data, display a first image in a first window on display 217 (the first image corresponds to the first image data), and display a third image in a second window on display 217 (the third image corresponds to the fused image data). In another aspect, the first image, the second image, and / or the third image may be displayed on the primary display and / or the secondary display. For example, a user may indicate that the primary display and / or the secondary display may display one of the first image, the second image, and the third image.

[0218] In one aspect, the present disclosure provides and illustrates a surgical communication and control headset that interfaces with the surgical hub 206 described in connection with FIGS. 1-11 . Further examples are disclosed in U.S. Patent Application Publication No. 2009 / 0046146, entitled "SURGICAL COMMUNICATION AND CONTROL SYSTEM," published February 19, 2009, and incorporated herein by reference in its entirety. FIG. 21 illustrates a schematic diagram of a beam source and associated beam detector system utilized as an equipment control mechanism in an operating room in accordance with at least one aspect of the present disclosure. For example, FIG. 21 illustrates a schematic diagram of a beam source and associated beam detector system utilized as an equipment control mechanism in an operating room. The system 6680 may be configured and wired to enable equipment control using overlays generated on the primary display (e.g., primary procedure display) and / or secondary display. The footswitch illustrates a method for allowing a user to click command icons that appear on the screen while the beam source is used to aim the particular desired command icon to be clicked. The beam source may also be used to indicate where the user is looking. The beam source may also be used by the user to indicate where data may be displayed. For example, the user may point the beam source at the primary display and / or the secondary display to indicate which display should be used to display the data.

[0219] A control system graphic user interface (GUI) communicates with the device control processor, and parameters are changed using the system. The system may include a display that may be coupled to a beam detection sensor. The display may be a primary display and / or a secondary display. For example, the system 6680 includes a display 6684 coupled to a beam detection sensor 6682. The system may include a head-mounted source 6686. The beam detection sensor 6682 may communicate with the control system GUI overlay processor and a beam source processor 6688. The surgeon may operate a foot switch 6692 or other attached switch, which provides a signal to the device control interface unit 6694.

[0220] The system 6680 may provide a means for sterile clinicians to control procedural equipment in an easy, fast, yet hands-free, centralized manner. The ability to maximize surgical efficiency and minimize the time the patient is under anesthesia is important for optimal patient outcomes. It is common for surgeons, cardiologists, or radiologists to verbally request adjustments to be made to specific medical devices and electronic equipment used in a procedure outside of the sterile field. It is common for them to have to rely on another staff member to make necessary adjustments to device settings, such as cameras, bobbies, surgical beds, shavers, inhalers, and injectors, to name a few. In many situations, having to command a staff member to change settings can slow the procedure because the non-sterile staff member is busy with another task. Because sterile physicians cannot adjust non-sterile equipment without compromising sterility, they often must wait for the non-sterile staff member to make the requested adjustments to specific devices before resuming the procedure.

[0221] The system 6680 allows the user to use the beam source and beam detector to recreate a pointer overlay coupled with the GUI, enabling a simultaneous switching method (i.e., footswitch, etc.) to allow the clinician to click commands on the primary and / or secondary displays. In one aspect, the GUI can appear on the procedural video display, which can be the primary and / or secondary display, when activated, such as when the user tilts their head twice to wake up or when they step on a footswitch provided with the system. Alternatively, a gesture such as a head tilt to the right can activate the system, while another gesture such as a head tilt to the left can simply activate the beam source. When the overlay (referred to as the device control GUI overlay) appears on the screen, it can show button icons representing various surgical devices, and the user can aim the button icons with the beam source, in this case a laser beam. When the laser is over the appropriate button icon, a footswitch or other simultaneous switching method can be activated, effectively acting like a mouse click on a computer. For example, a user can "wake up" the system to pop up a device control GUI overlay, which lists button icons on the screen, each labeled as a corresponding procedural medical device. The user may then aim the laser at the correct box or device and click the foot pedal (or some other parallel control, such as voice control, waistband button, etc.) to make a selection, similar to clicking a mouse on a computer. The sterilization physician can then select, for example, "inhaler." The next screen shows arrow icons that can be clicked for various settings (pressure, speed, etc.) of the device that need to be adjusted. In one iteration, the user can then aim the laser at the up arrow and click the foot pedal repeatedly until the desired setting is achieved.

[0222] In one aspect, a user, such as a sterilization physician, may use the beam to indicate where data may be displayed. For example, the user may be able to view the primary display and / or the secondary display. The user may desire to view contextual data, such as data related to an operation, on one or more of the displays. The user may use the beam to indicate that the contextual data should appear on the primary display. The user may use the beam to indicate that the contextual data should appear on the secondary display. The user may also use the beam to indicate that data from the primary display should be moved to the secondary display or that data should be moved from the secondary display to the primary display.

[0223] The surgical hub may provide interface controls with one or more primary displays and / or one or more secondary displays, which may be secondary surgeon display units. The primary and / or secondary displays may be designed to be within the sterile field.

[0224] Figures 22A-22E show various types of sterile field control and data entry consoles according to at least one embodiment of the present disclosure. Figure 22A shows a single-zone sterile field control and data entry console. Figure 22B shows a multi-zone sterile field control and data entry console. Figure 22C shows a tethered sterile field control and data entry console. Figure 22D shows a battery-powered sterile field control and data entry console. Figure 22E shows a battery-powered sterile field control and data entry console.

[0225] In one aspect, the surgical hub 206 may provide a secondary user interface that may allow for the display and control of the functions of the surgical hub 206 from the sterile field. The secondary display may be used to change the display location, what information is displayed where, and to hand off control of certain functions or devices. For example, the secondary display may be used by a user to move data display on the secondary display to the primary display. As another example, the secondary display may be used by a user to move data from the primary display to the secondary display. The secondary display may be internal to the medical instrument, external to the medical instrument, or associated with the medical instrument.

[0226] The display unit, which may be a primary display and / or a secondary display, may be designed for use within the sterile field and may be accessible for input and display by the surgeon to allow the surgeon to have interactive input control from the sterile field to control other surgical devices that may be coupled to the surgical hub. The display unit may be sterile and located within the sterile field, allowing the surgeon to interface with the display unit and the surgical hub to directly interface and configure instruments as needed without leaving the sterile field. The display unit may be used for display, control, and interchange of tool control, allowing the surgeon to feed from other surgical hubs without leaving the sterile field. The display unit may allow a user, such as a surgeon, to control the primary display and / or secondary display, which may be outside the sterile field. The display unit may allow a user to control the primary and / or secondary display, which may be within the sterile field.

[0227] In one aspect, the present disclosure provides a control unit comprising an interactive touchscreen display, an interface configured to couple the interactive touchscreen display to a surgical hub, a processor, and a memory coupled to the processor, the memory storing instructions executable by the processor to receive input commands from the interactive touchscreen display located inside the sterile field, transmit the input commands to the surgical hub, and control devices coupled to the surgical hub located outside the sterile field.

[0228] In one aspect, the present disclosure provides a control unit comprising an interactive touchscreen display, an interface configured to couple the interactive touchscreen display to a surgical hub, and control circuitry configured to receive input commands from the interactive touchscreen display located inside the sterile field, transmit the input commands to the surgical hub, and control devices coupled to the surgical hub located outside the sterile field.

[0229] A display unit may be provided that may be used within the sterile field and accessible for input and display by the surgeon. For example, the display unit may provide interactive input control to the surgeon from the sterile field to control other surgical devices coupled to the surgical hub.

[0230] This display unit within the sterile field is sterile and allows the surgeon to interface with it and the surgical hub. It gives the surgeon control of the instruments coupled to the surgical hub, allowing the surgeon to directly interface and configure instruments as needed without leaving the sterile field. The display unit may be used for display, control, tool control exchange, and feeds from other surgical hubs without the surgeon leaving the sterile field. For example, the display unit may be a primary and / or secondary display, and the display unit may be used to control the display of data on another primary and / or secondary display. In another example, the display unit may be used to move data displayed on one display to another display.

[0231] A secondary user interface may be used to enable display and control of the Surgical Hub functions from within the sterile field. This control may be a primary and / or secondary display and may be a display device such as an iPad, e.g., a portable interactive touchscreen display device configured to be introduced into the operating room in a sterile manner. It may be paired like any other device or may be location-sensitive. The display device may be enabled to function in this manner whenever it is placed over a location (e.g., a specific location). For example, the display device may be enabled to function in this manner whenever it is placed over the patient's draped abdomen during a surgical procedure.

[0232] In one aspect, the present disclosure provides a secondary user interface to enable viewing and control of surgical hub functions from within the sterile field. In one aspect, the secondary display can be used to change display location, determine what information is displayed where, and pass off control of certain functions or devices. For example, the secondary display may be used to transmit data displayed on the primary display.

[0233] There may be several different types of secondary surgical displays. For example, one type of secondary display may be designed for use within the sterile field and may be accessible for input and display by the surgeon within a sterile field interactive control display. The sterile field interactive control display may be a shared or common sterile field input control display. The sterile field display may be a primary display and / or a secondary display.

[0234] The sterile field display may be mounted on the operating table, on a stand, or simply placed on the patient's abdomen or chest. The sterile field display is sterile and allows the surgeon to interface with the sterile field display and surgical hub. This may give the surgeon control of the system and allow the surgeon to interface and configure the sterile field display as needed. The sterile field display may be configured as a master device and may be used for display, control, exchange of tool control, enabling feeds from other surgical hubs, etc. For example, the sterile field display may be a primary and / or secondary display and may allow the surgeon to control one or more primary and / or secondary displays.

[0235] In one aspect, the sterile field display can be employed to reconfigure wireless activation devices and their paired energy devices within the operating room (OR) when a surgeon hands off one device to another. Figures 22A-22E show various types of sterile field control and data entry consoles 6700, 6702, 6708, 6712, 6714 according to various aspects of the present disclosure. The disclosed sterile field control and data entry consoles 6700, 6702, 6708, 6712, 6714 each include at least one touchscreen 6701, 6704 / 6706, 6709, 6713, 6716 input / output device overlaid on top of an electronic visual display of an information processing system. The sterile field control and data entry consoles 6700, 6702, 6708, 6712, 6714 may include batteries as a power source. Some include a cable 6710 for connecting to a separate power source or for recharging the battery. A user can provide input or control the information processing system via simple or multi-touch gestures by touching the touchscreens 6701, 6704 / 6706, 6709, 6713, 6716 with a stylus, one or more fingers, or a surgical tool. The sterile field control and data entry consoles 6700, 6702, 6708, 6712, 6714 may be used to reconfigure wireless activation devices and paired energy devices in the operating room when a surgeon hands off the device to another surgeon. For example, the sterile field display may be a primary and / or secondary display and may allow the surgeon to control one or more primary and / or secondary displays.

[0236] The sterile field control and data entry consoles 6700, 6702, 6708, 6712, 6714 may be used to accept input from another operating room and would then configure some or all of the operating room screens to mirror the other operating room so that the surgeon can see what is needed to assist. The sterile field control and data entry consoles 6700, 6702, 6708, 6712, 6714 are configured to communicate with the surgical hub 206. As such, the description of the surgical hub 206 described in connection with Figures 1-11 is incorporated by reference into this section.

[0237] FIG. 22A shows a single-zone sterile field control and data entry console 6700 according to one embodiment of the present disclosure. The single-zone console 6700 is configured for use within a single zone within the sterile field. The single-zone console 6700 may be a secondary display. When deployed within the sterile field, the single-zone console 6700 can receive touchscreen input from a user within the sterile field. The touchscreen 6701 allows the user to directly interact with what is displayed rather than using a mouse, touchpad, or other such device (other than a stylus or surgical tool). The single-zone console 6700 includes wireless communication circuitry for wirelessly communicating to the surgical hub 206. The single-zone console 6700 may allow the user to control the primary display and / or another secondary display.

[0238] 22B shows a multi-zone sterile field control and data entry console 6702 according to one embodiment of the present disclosure. The multi-zone console 6702 includes a first touchscreen 6704 for receiving input from a first zone of the sterile field and a second touchscreen 6706 for receiving input from a second zone of the sterile field. The multi-zone console 6702 may be a secondary display. The multi-zone console 6702 is configured to receive input from multiple users within the sterile field. The multi-zone console 6702 includes wireless communication circuitry for wirelessly communicating to the surgical hub 206. Thus, the multi-zone sterile field control and data entry console 6702 includes an interactive touchscreen display with multiple input and output zones. The multi-zone console 6702 may allow a user to control the primary display and / or a separate secondary display.

[0239] 22C shows a tethered sterile field control and data entry console 6708 according to one embodiment of the present disclosure. The tethered console 6708 includes a cable 6710 for connecting the tethered console 6708 to the surgical hub 206 via a wired connection. The cable 6710 allows the tethered console 6708 to communicate via a wired link in addition to a wireless link. The cable 6710 also allows the tethered console 6708 to connect to a power source for powering the console 6708 and / or recharging batteries within the console 6708. The tethered console 6708 may be a secondary display. The tethered console 6708 may allow a user to control the primary display and / or another secondary display.

[0240] FIG. 22D shows a battery-powered sterile field control and data entry console 6712 according to one embodiment of the present disclosure. The sterile field console 6712 is battery-powered and includes wireless communication circuitry for wirelessly communicating with the surgical hub 206. In one embodiment, the sterile field console 6712 can be configured to communicate with any of the modules coupled to the hub 206, such as the generator module 240. Through the sterile field console 6712, the surgeon may adjust the generator's power output level using a touchscreen 6713 interface. An example is described below in connection with FIG. 22E. The sterile field console 6712 can be a secondary display. The sterile field console 6712 can allow a user to control the primary display and / or another secondary display.

[0241] FIG. 22E illustrates a battery-powered sterile field control and data entry console 6714 according to one embodiment of the present disclosure. The sterile field console 6714 may include a user interface displayed on the generator's touchscreen. The surgeon may therefore control the generator's output by touching the up / down arrow icons 6718A, 6718B, which increase / decrease the generator module 240's power output. Additional icons 6719 allow access to generator module settings 6174, volume 6178, among other features, using the + / - icon directly from the sterile field console 6714. The sterile field console 6714 may also be employed to adjust or reconfigure settings of other wireless activation devices or modules coupled to the hub 206 in the operating room and their paired energy devices when the surgeon hands off the sterile field console 6714 to another. The sterile field console 6714 may be a secondary display. The sterile field console 6714 may allow the user to control the primary display and / or another secondary display.

[0242] 23A-23B illustrate a sterile field console 6700 for use in a sterile field during a surgical procedure, according to one embodiment of the present disclosure. FIG. 23 illustrates a sterile field console 6714 positioned in the sterile field near two surgeons performing the procedure. In FIG. 23, one of the surgeons is shown tapping the sterile field console's touchscreen 6701 with a surgical tool 6722 to adjust the output of a modular device coupled to the surgical hub 206 and to reconfigure the modular device or an energy device paired with a modular device coupled to the surgical hub 206.

[0243] The sterile field display may be employed as an interactive scalable secondary display that allows the surgeon to overlay other feeds or images, such as a laser Doppler scanning array. In one aspect, the sterile field display may be employed to recall pre-operative scans or images for review. Once the vessel path and depth and device trajectory are estimated, the surgeon employs a sterile field interactive scalable secondary display that allows the surgeon to overlay other feeds or images.

[0244] 24 is a diagram 6770 illustrating a technique for estimating vessel path, depth, and device trajectory. Using a standard approach, before cutting blood vessels 6772, 6774 located below the surface of tissue 6775, a surgeon estimates the path and depth of the blood vessels 6772, 6774 and the trajectory 6776 that a surgical device 6778 will take to reach the blood vessels 6772, 6774. It is often difficult to estimate the path and depth 6776 of blood vessels 6772, 6774 located below the surface of tissue 6775 because the surgeon cannot precisely visualize the location of the path and depth 6776 of the blood vessels 6772, 6774.

[0245] 25A-25D show multiple real-time views of images of virtual anatomical details for dissection, including perspective (FIGS. 25A, 25C) and side (FIGS. 25B, 25D) views. The images may be displayed on a primary display and / or a secondary display. For example, the images may be displayed on a sterile field display of a tablet computer or sterile field control and data entry console, employed as an interactive, scalable secondary display that allows the surgeon to overlay other feeds or images, according to one aspect of the present disclosure. The images of the virtual anatomy may enable the surgeon to more accurately predict the path and depth of vessels 6772, 6774 located below the surface of the tissue 6775, as shown in FIG. 24, and the best trajectory 6776 of the surgical device 6778.

[0246] FIG. 25A is a perspective view of a virtual anatomy 6780 displayed on a secondary device, such as a tablet computer or sterile field control and data entry console. FIG. 25B is a side view of the virtual anatomy 6780 shown in FIG. 25A , according to one embodiment of the present disclosure. Referring to FIGS. 25A-25B , in one embodiment, a surgeon uses a smart surgical device 6778 and a tablet computer to visualize the virtual anatomy 6780 in real time and in multiple views. The smart surgical device 6778 may include a display, which may be a secondary display. The tablet computer may include a display, which may be a primary display and / or a secondary display. The three-dimensional perspective view includes a portion of tissue 6775 with blood vessels 6772, 6774 located below the surface. The portion of tissue is overlaid with a grid 6786 to allow the surgeon to visualize scale and measure the path and depth of the blood vessels 6772, 6774 at target locations 6782, 6784, each marked with an X. The grid 6786 also assists the surgeon in determining the best trajectory 6776 for the surgical device 6778. As shown, the blood vessels 6772, 6774 have an aberrant vascular pathway.

[0247] FIG. 25C shows a perspective view of a virtual anatomy 6780 for dissection, according to one embodiment of the present disclosure. FIG. 25D shows a side view of a virtual anatomy 6780 for dissection, according to one embodiment of the present disclosure. With reference to FIGS. 25C-25D, using a tablet computer, a surgeon can zoom and pan 360-∞ to obtain an optimal view of the virtual anatomy 6780 for dissection. The surgeon then determines the best path or trajectory 6776 for inserting a surgical device 6778 (e.g., a dissection instrument, in this example). The surgeon can view the anatomical structure in a three-dimensional perspective view or in any one of six views. See, for example, the side view of the virtual anatomy 6780 and the insertion of a surgical device 6778 (e.g., a dissection instrument) in FIG. 25D.

[0248] In another aspect, the sterile field control and data entry console may allow live chat between different departments, such as, for example, the oncology or pathology departments, to discuss margins or other details associated with imaging. The sterile field control and data entry console may allow the pathology department to communicate with the surgeon about the relationship of margins within the specimen and show them to the surgeon in real time using the sterile field console.

[0249] In another aspect, the sterile field control and data entry console may be used to change the focus and field of view of its own image or to control any of the other monitors coupled to the surgical hub. For example, the sterile field control and data entry console may be a primary and / or secondary display that may be used to control another primary and / or secondary display.

[0250] In another aspect, the sterile field control and data entry console can be used to display the status of any of the instruments or modules coupled to the surgical hub 206. Knowledge of which devices coupled to the surgical hub 206 are in use can be obtained through information such as the device not being on an instrument pad or on-device sensors. Based on this information, the sterile field control and data entry console can change the display, configuration, switch power to power one device and not power another, switch one code from the instrument pad to multiple codes from the instrument pad, and so on. Device diagnostics can obtain knowledge that a device is inactive or not in use. Device diagnostics can be based on information such as the device not being on an instrument pad or based on sensors on the device.

[0251] In another aspect, the sterile field control and data entry console may be used as a learning tool. The console may display checklists, procedure steps, and / or sequences of steps. A timer / clock may be displayed to measure the time to complete steps and / or procedures. The console may display room sound pressure levels as an indicator of activity, stress, etc.

[0252] 26A-26E show a touchscreen display 6890 that may be used in the sterile field according to one embodiment of the present disclosure. The touchscreen display 6890 may be a primary display and / or a secondary display. Using the touchscreen display 6890, a surgeon may manipulate an image 6892 displayed on the touchscreen display 6890 using various gestures, such as, for example, drag and drop, scroll, zoom, rotate, tap, double tap, flick, drag, swipe, pinch open, pinch close, touch and hold, two-finger scroll, among others. Using the touchscreen display 6890, a surgeon may manipulate an image 6892, which may be displayed on another primary display and / or a secondary display, using various gestures, such as, for example, drag and drop, scroll, zoom, rotate, tap, double tap, flick, drag, swipe, pinch open, pinch close, touch and hold, two-finger scroll, among others. The surgeon may also use gestures, such as gestures on the touchscreen display 6890, to move images or data displayed on the touchscreen display 6890 to another primary and / or secondary display. The surgeon may also use gestures, such as gestures on the touchscreen display 6890, to move images or data displayed on the primary and / or secondary display to the touchscreen display 6890.

[0253] Figure 26A shows an image 6892 of a surgical site displayed on a touchscreen display 6890 in portrait mode. Figure 26B shows the touchscreen display 6890 rotated to landscape mode (e.g., arrow 6894), with the surgeon using his index finger 6896 to scroll the image 6892 in the direction of the arrow. Figure 26C shows the surgeon using his index finger 6896 and thumb 6898 to pinch open and zoom out the image 6892 in the direction of arrow 6899. Figure 26D shows the surgeon using his index finger 6896 and thumb 6898 to pinch open and zoom out the image 6892 in the direction of arrow 6897. Figure 26E shows the touchscreen display 6890 rotated in two directions indicated by arrows 6894, 6896 so that the surgeon can view the image 6892 in different orientations.

[0254] Control and static displays are used outside the sterile field that may differ from the control and static displays used inside the sterile field. Control and static displays located outside the sterile field provide interactive and static displays for operating room (OR) and equipment control. Control and static displays located outside the sterile field may be primary and / or secondary displays. Control and static displays located outside the sterile field may include secondary displays, such as secondary static displays and secondary touchscreens, for input and output.

[0255] Non-sterile displays 107, 109, 119 (FIG. 2) may be used outside the sterile field and may include monitors located on walls of operating rooms, on rotating stands, or on capital equipment. The displays may be presented with a feed from the control devices to which they are attached and may display what is presented to them.

[0256] The one or more secondary displays, which may be secondary touch input screens located outside the sterile field, may be part of the visualization system 108 (FIG. 2), part of the surgical hub 106 (FIG. 2), or may be fixed-position touch monitors on a wall or rotating stand. The difference between a touch input screen and a static display may be that the user may interact with the touch input screen by changing what may be displayed on that particular monitor or other. In capital equipment applications, it may be an interface for controlling settings of the connected capital equipment. The primary and / or secondary displays outside the sterile field may be used to preload surgeon preferences. For example, touch input screens and static displays outside the sterile field may be used to preload surgeon preferences (instrumentation settings and modes, lighting, procedures and preferred steps and sequences, music, etc.).

[0257] A secondary display, such as a secondary surgeon display, may function similarly to a sterile field input display device but may include a personal input display with a personal input device that can be controlled by the surgeon. Secondary displays, such as a personal secondary display, may be implemented in many form factors, such as a watch, a small display pad, or interface glasses. A personal secondary display may include control capabilities for a display device and may be located on the surgeon or controlled by the surgeon. A personal secondary display may be keyed to the surgeon (e.g., specifically keyed to the surgeon) and may indicate to one or more users, itself, one or more primary displays, one or more secondary displays, and / or other devices. A personal secondary display may be used to grant permission to release a device. A personal secondary display may be used to control one or more primary and / or secondary displays. For example, a personal secondary display may be used to control what is displayed on the primary and / or secondary displays. As another example, a personal secondary display may be used to move data from one display to another.

[0258] A personal secondary display can be used to provide data dedicated to one of multiple surgeons who wants to monitor something they don't want others to monitor. A personal secondary display may also be used as a command module. A personal secondary display may be held by the lead surgeon in the operating room and give the surgeon control to override any other input from anyone else. A personal secondary display may be coupled to a short-range wireless (e.g., Bluetooth) microphone and / or earphones, allowing the surgeon to have a separate conversation or call, or the personal secondary display may be used to broadcast to all others in the operating room or other department. The surgeon may also issue verbal commands to the personal secondary display using the microphone and / or earphones. The surgeon may also use gestures to provide one or more commands to the personal secondary display.

[0259] 27 is a process logic flow diagram 6920 illustrating a control program or logic configuration for communicating from inside the sterile field to devices located outside the sterile field, according to one embodiment of the present disclosure. In one embodiment, the control unit may include an interactive touchscreen display, an interface configured to couple the interactive touchscreen display to a surgical hub, a processor, and memory coupled to the processor. The memory may store instructions executable by the processor to receive 6922 input commands from the interactive touchscreen display located inside the sterile field, send the input commands to the surgical hub, and control 6924 devices coupled to the surgical hub located outside the sterile field.

[0260] 28 shows a second information layer overlaid on the first information layer. The second information layer includes a symbolic representation of a knife that overlaps the detected location of the knife in a disposable loading unit (DLU) shown on the first information layer. Further examples are disclosed in U.S. Patent No. 9,283,054, entitled "SURGICAL APPARATUS WITH INDICATOR," issued March 15, 2016, which is incorporated herein by reference in its entirety.

[0261] 28 , the second information layer 6963 can overlay at least a portion of the first information layer 6962 on the display 6960. Additionally, a touchscreen 6961, which may be a primary display and / or a secondary display, can allow a user to manipulate the second information layer 6963 relative to video feedback in the underlying first information layer 6962 on the display 6960. For example, a user can manipulate the touchscreen 6961 to select, manipulate, reformat, resize, and / or otherwise modify information displayed on the second information layer 6963. In one aspect, a user can move the first information layer and / or the second information layer to one or more displays, which may include a primary display and / or a secondary display. In one aspect, a user can manipulate the touchscreen 6961 to manipulate the second information layer 6963 relative to a surgical instrument 6964 shown in the first information layer 6962 on the display 6960. The user may, for example, select a menu, category, and / or classification of the control panel 6967, and the second information layer 6963 and / or control panel 6967 may adjust to reflect the user's selection. In various aspects, the user may select a category from the instrument feedback categories 6969 that corresponds to a particular feature(s) of the surgical instrument 6964 shown in the first information layer 6962. The feedback corresponding to the user-selected category may move, position itself, and / or "snap" to a location on the display 6960 relative to the particular feature(s) of the surgical instrument 6964. For example, the selected feedback may move to a location near and / or overlapping with a particular feature or features of the surgical instrument 6964 shown on the first information layer 6962.

[0262] The instrument feedback menu 6969 may include multiple feedback categories and may relate to feedback data measured and / or detected by the surgical instrument 6964 during a surgical procedure. As described herein, the surgical instrument 6964 can detect and / or measure, for example, the position 6970 of the movable jaw between an open and a closed orientation, the thickness 6973 of the clamped tissue, the clamping force 6976 on the clamped tissue, the articulation 6974 of the DLU 6965, and / or the position 6971, velocity 6972, and / or force 6975 of the firing element. Additionally, a feedback controller in signal communication with the surgical instrument 6964 can provide sensed feedback to the display 6960, which can display the feedback on the second information layer 6963. As described herein, the selection, arrangement, and / or format of the feedback data displayed in the second information layer 6963 can be changed based on, for example, user input to the touchscreen 6961.

[0263] For example, when the knife of DLU6965 is obscured from view by the end effector jaw 6966 and / or tissue T, the operator may track and / or estimate the position of the knife within DLU6965 based on the changing values ​​of the feedback data and / or the shifting position of the feedback data relative to DLU6965 as shown in the underlying first information layer 6962.

[0264] In various aspects, the display menu 6977 of the control panel 6967 may be associated with multiple categories, such as, for example, a unit system 6978 and / or a data mode 6979. In particular aspects, a user can select the unit system category 6978 to switch between unit systems, such as, for example, between metric units and U.S. customary units. Additionally, a user can select the data mode category 6979 to switch, for example, between the type of numeric representation of the feedback data and / or the type of graphical representation of the feedback data. The numeric representation of the feedback data can be displayed, for example, as a number and / or a percentage. Furthermore, the graphical representation of the feedback data can be displayed, for example, as a function of time and / or distance. As described herein, a user can select an instrument controller menu 6980 from the control panel 6967 to input instructions for the surgical instrument 6964, which can be implemented, for example, via the instrument controller and / or microcontroller. The user can minimize or collapse the control panel 6967 by selecting the minimize / maximize icon 6968, and can maximize or expand the control panel 6967 by reselecting the minimize / maximize icon 6968.

[0265] FIG. 29 shows a perspective view of a surgeon using a surgical instrument including a handle assembly housing and a wireless circuit board during a surgical procedure, with the surgeon wearing safety glasses. The wireless circuit board transmits signals to safety glasses worn by the surgeon using the surgical instrument during the procedure. The signals are received by a wireless port on the safety glasses. One or more lighting devices on the front lenses of the safety glasses change color, fade, or glow in response to the received signals to provide the surgeon with information regarding the status of the surgical instrument. The lighting devices can be positioned on the periphery of the front lens so as not to obstruct the surgeon's direct line of sight. Further examples are disclosed in U.S. Patent No. 9,011,427, entitled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued April 21, 2015, the entire contents of which are incorporated herein by reference.

[0266] 29 shows a version of safety glasses 6991 that may be worn by a surgeon 6992 during a surgical procedure while using a medical device. The safety glasses 6991 may be a primary display and / or a secondary display. The safety glasses 6991 may be used to determine the direction the surgeon 6992 is looking. For example, the safety glasses 6991 may analyze the movement of the surgeon's 6992's pupils (e.g., using an internal or external camera) and determine that the surgeon is looking at a monitor 6997. As another example, the safety glasses 6991 may track the movement of the surgeon's head using one or more sensors to determine where the surgeon is looking (e.g., whether the surgeon is looking at a monitor 6997).

[0267] During use, a wireless communication board housed within the surgical instrument 6993 can communicate with a wireless port 6994 on the safety glasses 6991. The exemplary surgical instrument 6993 is a battery-powered device, although the instrument 6993 can be powered by cable or other methods. The instrument 6993 includes an end effector. In particular, the wireless communication board (6995) transmits one or more wireless signals, indicated by arrows (B, C), to the wireless port (6994) on the safety glasses (6991). The safety glasses 6991 receive the signals, analyze the received signals, and display status information indicated by the signals on the lenses 6996 to a user, such as a surgeon 6992, wearing the safety glasses 6991.

[0268] The wireless communication board 6995 can transmit wireless signals to the surgical monitor 6997 so that the surgical monitor 6997 can display the received and indicated status information to the surgeon 6992, as described herein. The surgical monitor 6997 can be a primary display and / or a secondary display.

[0269] Versions of the safety glasses 6991 may include lighting devices on the periphery of the safety glasses 6991. The lighting devices provide peripheral vision sensory feedback of the device 6993 by which the safety glasses 6991 communicate to a user wearing the safety glasses 6991. The lighting devices may be, for example, light-emitting diodes ("LEDs"), a series of LEDs, or any other suitable lighting devices known to those skilled in the art and apparent in view of the teachings herein.

[0270] The LED may be located on the edge or side of the front lens of the safety glasses 6991 so that it is still positioned within the user's field of view, but not off-center, so that the user does not have to look away from the surgical site to see the illumination device. The displayed light may flash and / or change color to communicate various aspects of information read from the instrument 6993 to the wearer of the safety glasses 6991, such as system status information or tissue sensing information (i.e., whether the end effector has sufficiently cut and sealed the tissue). Feedback from the housed wireless communication board 6995 may activate, flash, or change color of the illumination device to indicate information to the user regarding the use of the instrument 6993. For example, the device may incorporate a feedback mechanism based on one or more sensed tissue parameters. In this case, a change in device output based on this feedback synchronized with a state change can signal the safety glasses 6991 via the wireless communication board 6995 to trigger activation of the illumination device. Such described means of activating the illumination device should not be considered limiting, as other means of indicating instrument 6993 status information to a user via the safety glasses 6991 are contemplated. Furthermore, the safety glasses 6991 may be disposable or reusable eyewear. A button-type power supply, such as a button-type battery, may be used to power the wireless receiver and LEDs of various versions of the safety glasses (6991), which may also include an enclosed wireless board and tri-color LED. In use, such a button-type power supply can provide a low-cost means of providing sensory feedback of information regarding the instrument (6993) to a surgeon (6992) wearing the safety glasses (6991).

[0271] It is an unfortunate reality that the outcomes of all surgical procedures are not always optimal and / or successful. For example, when a fault event is detected and / or identified, communication methods can be utilized to separate surgical data that may be associated with the fault event (e.g., fault event surgical data) from surgical data that may not be associated with the fault event (e.g., non-fault event surgical data), and the surgical data that may be associated with the fault event (e.g., fault event data) can be communicated from the surgical hub 206 to the cloud-based system 205 on a priority basis for analysis. According to one aspect of the present disclosure, the fault event surgical data can be communicated from the surgical hub 206 to the cloud-based system 205 in priority to the non-fault event surgical data.

[0272] FIG. 30 illustrates various aspects of a system-implemented method for identifying surgical data associated with a fault event (e.g., fault event surgical data) and communicating the identified surgical data to the cloud-based system 205 on a prioritized basis. The method includes receiving (3838) surgical data at the surgical hub 206, where the surgical data is associated with a surgical procedure; time-stamping (3840) the surgical data; identifying a fault event associated with the surgical procedure (3842); determining which of the surgical data is associated with the fault event (e.g., fault event surgical data) (3844); separating (3846) the surgical data associated with the fault event from all other surgical data received at the surgical hub 206 (e.g., non-fault event surgical data); time-seriesing (3848) the surgical data associated with the fault event; encrypting (3850) the surgical data associated with the fault event; and communicating (3852) the encrypted surgical data to the cloud-based system 205 based on a priority.

[0273] More specifically, various surgical data may be captured during a surgical procedure, and the captured surgical data, as well as other surgical data associated with the surgical procedure, may be communicated to the surgical hub 206. The surgical data may include, for example, data associated with the surgical devices / instruments utilized during the procedure (e.g., FIG. 5 , surgical devices / instruments 235), data associated with the patient, data associated with the facility where the surgical procedure was performed, and data associated with the surgeon. Either before or after the surgical data is communicated to and received by the surgical hub 206, the surgical data may be time-stamped and stripped of any information that may identify a particular procedure, patient, or surgeon, such that the information is essentially anonymized for further processing and analysis by the cloud-based system 205.

[0274] When a fault event is detected and / or identified (e.g., which may be either during or after the surgical procedure), the surgical hub 206 may determine which of the surgical data is associated with the fault event (e.g., fault event surgical data) and which of the surgical data is not associated with the surgical event (e.g., non-fault event surgical data). According to one aspect of the present disclosure, a fault event may include, for example, the detection of one or more misfired staples during the stapling portion of the surgical procedure. For example, in one aspect, with reference to FIG. 5 , the endoscope 239 may take snapshots while the surgical device / instrument 235 comprising an end effector including a staple cartridge performs the stapling portion of the surgical procedure. In one such aspect, the imaging module 238 may compare the snapshots with stored images and / or images downloaded from the cloud-based system 205 depicting correctly fired staples to detect misfired staples and / or evidence of misfired staples (omissions). In another aspect, the imaging module 238 may analyze the snapshot itself to detect misfired staples and / or evidence of misfired staples. In an alternative aspect, the surgical hub 206 may communicate the snapshot to the cloud-based system 205, where components of the cloud-based system 205 may perform the various imaging module functions described above to detect misfired staples and / or evidence of misfired staples and report the detection to the surgical hub 206. According to another aspect of the present disclosure, a fault event may include detection of tissue temperature being below an expected temperature during the tissue sealing portion of the surgical procedure and / or a visual indication (e.g., via endoscope 239, FIG. 5 ) of excessive bleeding or oozing after the surgical procedure. 5, the surgical device / instrument 235 may include an end effector including a temperature sensor and the surgical hub 206, and / or the cloud-based system may compare at least one temperature detected by the temperature sensor (e.g., during the tissue sealing portion of the surgical procedure) to a stored temperature and / or temperature range expected and / or associated with the surgical procedure to detect improper / low sealing temperatures. In another aspect, the endoscope 239 may take snapshots during the surgical procedure.In one such aspect, the imaging module 238 may compare the snapshot with stored images and / or images downloaded from the cloud-based system 205 that convey tissue that was sealed correctly at the expected temperature to detect evidence of an improper / insufficient seal temperature (e.g., charring, oozing / bleeding). Further, in one such aspect, the imaging module 238 may analyze the snapshot itself to detect evidence of an improper / insufficient seal temperature (e.g., charring, oozing / bleeding). As another example, the surgical hub 206 may communicate the snapshot to the cloud-based system 205, and components of the cloud-based system 205 may perform the various imaging module functions described above to detect evidence of an improper / insufficient seal temperature and report that detection to the surgical hub 206. According to various aspects described herein, in response to a detected and / or identified fault event, the surgical hub 206 may download a program from the cloud-based system 205 for execution by the surgical device / instrument 235 that corrects the detected problem (e.g., a program that alters surgical device / instrument parameters to prevent misfired staples, a program that alters surgical device / instrument parameters to ensure accurate sealing temperatures).

[0275] In some aspects, a fault event may be considered to encompass a particular time period, and one or more (e.g., all) surgical data associated with that time period may be considered to be associated with the fault event.

[0276] After the surgical data associated with the fault event has been identified, the identified surgical data (e.g., fault event surgical data) may be separated or isolated from some or all of the other surgical data associated with the surgical procedure (e.g., non-fault event surgical data). This separation can be achieved, for example, by tagging or flagging the identified surgical data, by storing the identified surgical data separately from all of the other surgical data associated with the surgical procedure, or by continuing to process the identified surgical data for subsequent prioritized communication to the cloud-based system 205 while storing only the other surgical data. According to various aspects, tagging or flagging the identified surgical data can occur during the communication process when the datagrams are generated, as described in more detail below.

[0277] A timestamp of the surgical data (e.g., either before or after the surgical data is received at the surgical hub) may be utilized by a component of the surgical hub 206 to time-series the identified surgical data associated with the failure event. The component of the surgical hub 206 utilizing the timestamp to time-series the identified surgical data may be, for example, the processor module 232, the processor 244 of the computer system 210, and / or a combination thereof. By time-seriesing the identified surgical data, the cloud-based system 205 and / or other parties may subsequently better understand the conditions leading to the occurrence of the failure event and perhaps pinpoint the exact cause of the failure event, thereby providing knowledge to potentially mitigate the occurrence of similar failure events during similar surgical procedures performed in the future.

[0278] If the identified surgical data is time-serialized, the time-serialized surgical data may be encrypted in a manner similar to that described above with respect to the encryption of generator data. Thus, the identified surgical data may be encrypted to help ensure the confidentiality of the identified surgical data either while stored in the surgical hub 206 or while being transmitted to the cloud-based system 205 using the Internet or other computer network. According to various aspects, a component of the surgical hub 206 utilizes an encryption algorithm to convert the identified surgical data from a readable version to an encoded version, thereby forming encrypted surgical data associated with the failure event. The component of the surgical hub utilizing the encryption algorithm may be, for example, the processor module 232, the processor 244 of the computer system 210, and / or a combination thereof. The encryption algorithm utilized may be a symmetric encryption algorithm or an asymmetric encryption algorithm.

[0279] After the identified surgical data is encrypted, a component of the surgical hub may communicate the encrypted surgical data associated with the failure event (e.g., encrypted failure event surgical data) to the cloud-based system 205. The component of the surgical hub that communicates the encrypted surgical data to the cloud-based system 205 may be, for example, the processor module 232, the hub / switch 207 / 209 of the modular communications hub 203, the router 211 of the modular communications hub 203, or the communications module 247 of the computer system 210. According to various aspects, communication of the encrypted surgical data (e.g., encrypted failure event surgical data) over the Internet may provide datagrams containing the encrypted surgical data to be delivered and may follow IP, which may provide an addressing method used to label the datagrams with source and destination information. The datagrams may include a field containing a flag or tag that identifies the encrypted surgical data (e.g., encrypted failure event surgical data) as prioritized relative to other unprioritized surgical data (e.g., encrypted non-failure event surgical data).

[0280] In some aspects, once a failure event associated with a surgical procedure is identified, the surgical hub 206 and / or cloud-based system 205 may subsequently flag or tag the surgical device / instrument 235 utilized during the surgical procedure for inoperability and / or removal. For example, in one aspect, information associated with the surgical device / instrument 235 and stored in the surgical hub 206 and / or cloud-based system 205 (e.g., serial number, ID) may be utilized to effectively prevent (e.g., blacklist) the surgical device / instrument 235 from being used again. In another aspect, information associated with the surgical device / instrument (e.g., serial number, ID) may initiate the printing of a shipping slip and shipping instructions to return the surgical device / instrument 235 to the manufacturer or other designated party so that a thorough analysis / inspection of the surgical device / instrument 235 can be performed (e.g., to determine the cause of the failure). According to various aspects described herein, once the cause of the failure is determined (e.g., via the surgical hub 206 and / or the cloud-based system 205), the surgical hub 206 may download from the cloud-based system 205 a program for execution by the surgical device / instrument 235 that corrects the determined cause of the failure (i.e., a program that alters surgical device / instrument parameters to prevent the failure from occurring again).

[0281] In some aspects, the primary display and / or secondary display may be used to provide or display a notification that an operational error has occurred. For example, when a fault event associated with a surgical procedure is identified, the surgical hub 206 and / or cloud-based system 205 may transmit an error message that is displayed on one or more primary and / or secondary displays. The error message may indicate to the user that a fault event has occurred, provide instructions to correct the error, provide recommendations for correcting the error, or provide instructions that may modify the surgical procedure. For example, an error message on the primary display may provide instructions for a surgical error that may have occurred to the patient due to the fault event. As another example, an error message on the secondary display may provide instructions to the user on how to remove a misfired staple and reload the staple cartridge.

[0282] According to some aspects, the surgical hub 206 and / or cloud-based system 205 may also provide / display reminders (e.g., via the hub display 215 and / or the surgical device / instrument display 237) to administrators, personnel, and / or other interested parties to physically remove the surgical device / instrument 235 from the operating room (e.g., if it is detected as still present in the operating room) and / or to transmit the surgical device / instrument 235 to the manufacturer or other designated party. In one aspect, the reminders may be set to be provided / displayed periodically until the administrator may remove the flag or tag for the surgical device / instrument 235 from the surgical hub 206 and / or cloud-based system 205. According to various aspects, the administrator may remove the flag or tag once the administrator may confirm that the surgical device / instrument 235 has been accepted by the manufacturer or other designated party (e.g., system tracking of the surgical device / instrument 235 via its serial number / ID). By using the methods described herein to flag and / or track surgical data associated with fault events, closed-loop control of surgical data associated with fault events and / or surgical devices / instruments 235 may be achieved. It will be appreciated that the surgical hub 206 may be utilized to effectively manage the utilization (or non-utilization) of surgical devices / instruments 235 utilized or potentially utilized during a surgical procedure.

[0283] In various aspects of the present disclosure, the surgical hub 206 and / or cloud-based system 205 may desire to control which components (e.g., surgical devices / instruments 235, energy devices 241) are utilized within its interactive surgical system 100 / 200 to perform a surgical procedure (e.g., to avoid the use of unauthorized or knock-off components to minimize future failure events).

[0284] Thus, in various aspects of the present disclosure, because the interactive surgical system 100 may include multiple surgical hubs 106, the cloud-based system 105 of the interactive surgical system 100 and / or each surgical hub 106 may wish to track the component-surgical hub combinations utilized over time. In one aspect, when / after a component (see FIG. 5 , e.g., surgical device / instrument 235, energy device 241) is connected to / used with a particular surgical hub 106 (e.g., surgical device / instrument 235 connected wired / wirelessly to a particular surgical hub 106, energy device 241 connected to a particular surgical hub 106 via generator module 240), the particular surgical hub 106 may communicate a record / block of that connection / usage to the cloud-based system 105 of the interactive surgical system 100 and / or other surgical hubs 106 (e.g., linking the unique identifiers of each of the connected devices). For example, upon / after connection / use of the energy device 241, a particular surgical hub 106 may communicate the record / block to the cloud-based system 105 of the interactive surgical system 100 and / or other surgical hubs 106 (e.g., linking the unique identifier of the energy device 241 to the unique identifier of the generator module 240 to the unique identifier of the particular surgical hub 106). In one such aspect, if this is the first time a component (e.g., an energy device) has been connected to / used with a surgical hub 106 of the interactive surgical system 100, the cloud-based system 105 and / or each surgical hub 106 of the interactive surgical system 100 may store the record / block as an occurrence record / block. In one such aspect, the occurrence record / block stored in the cloud-based system 105 and / or each surgical hub 106 may include a timestamp.However, in one such aspect, if this was not the first time the component (e.g., energy device 241) was connected to / used with a surgical hub 106 of the interactive surgical system 100, the cloud-based system 105 and / or each surgical hub 106 of the interactive surgical system may store the record / block as a new record / block in the series of records / blocks associated with the component. In such an aspect, the new record / block may include a cryptographic hash of the most recently communicated record / block, the communicated link data, and a timestamp stored in the cloud-based system 105 and / or each surgical hub 106. In one such aspect, each cryptographic hash links each new record / block (e.g., each use of the component) to its previous record / block, forming a chain that confirms the integrity of each previous record / block back to the original originating record / block (e.g., the first use of the component). According to one such aspect, this blockchain of records / blocks may be deployed at the cloud-based system 105 and / or each surgical hub 106 of the interactive surgical system 100 to permanently and verifiably tie the use of a particular component over time to one or more surgical hubs 106 of the interactive surgical system 100. Here, according to another aspect, this approach may similarly be applied to sub-components of a component (e.g., handle, shaft, end effector, cartridge) as / after the component is connected to / used with a particular surgical hub 106 of the interactive surgical system 100.

[0285] According to various aspects of the present disclosure, the cloud-based system 105 and / or each surgical hub 106 may utilize such records / blocks to track the use of a particular component and / or sub-component back to its first use in the interactive surgical system 100. For example, if a particular component (e.g., a surgical device / instrument 235) is flagged / tagged in association with a fault event, the cloud-based system 105 and / or surgical hub 106 may analyze such records / blocks to determine whether past use of that component and / or its sub-components contributed to or caused the fault event (e.g., overuse). In one example, the cloud-based system 105 may determine that a sub-component of that component (e.g., an end effector) may indeed have contributed to / caused the fault event and then tag / flag the component for inoperability and / or removal based on the determination.

[0286] According to another aspect, the cloud-based system 205 and / or the surgical hub 206 may control which components (e.g., surgical devices / instruments 235, energy devices 241) are utilized in the interactive surgical system 200 to perform a surgical procedure by authenticating the components and / or their suppliers / manufacturers. In one aspect, the supplier / manufacturer of a component may associate a serial number and a source ID with the component. In one such aspect, the supplier / manufacturer may create / generate a private key for the serial number, encrypt the serial number using the private key, and store the encrypted serial number and source ID on an electronic chip (e.g., memory) within the component before shipping it to the surgical site. Then, upon / after connecting the component to the surgical hub 206, the surgical hub 206 may read the encrypted serial number and source ID from the electronic chip. In response, the surgical hub 206 may send a message (i.e., including the encrypted serial number) to the supplier / manufacturer's server associated with the source ID (e.g., directly or via the cloud-based system 205). In one such aspect, the surgical hub 206 may encrypt a message using a public key associated with its supplier / manufacturer. In response, the surgical hub 206 may receive a message from the supplier / manufacturer server (e.g., directly or via the cloud-based system 205) (i.e., including a private key that the supplier / manufacturer generated for / associated with the encrypted serial number). In one such aspect, the supplier / manufacturer server may encrypt the message using a public key associated with the surgical hub 206. Further, in one such aspect, the surgical hub 206 may then decrypt the message (e.g., using a private key paired to the public key used to encrypt the message) to reveal the private key associated with the encrypted serial number. The surgical hub 206 may then decrypt the encrypted serial number using the private key to reveal the serial number.Further, in one such aspect, the surgical hub 206 may then compare the decrypted serial number to a comprehensive list of authorized serial numbers (e.g., stored in the surgical hub 206 and / or in a cloud-based system and / or downloaded from the cloud-based system, e.g., separately received from the supplier / manufacturer) and may authorize use of the connected component if the decrypted serial number matches an authorized serial number. Initially, such a process allows the surgical hub 206 to authenticate the supplier / manufacturer. Specifically, the surgical hub 206 encrypted a message including the encrypted serial number using a public key associated with the supplier / manufacturer. Thus, receiving a response message (i.e., including the private key) authenticates the supplier / manufacturer to the surgical hub 206 (i.e., otherwise, the supplier / manufacturer would not have access to the private key paired to the public key used by the surgical hub 206 to encrypt the message, and the supplier / manufacturer would not have been able to associate the encrypted serial number received in the message with its already-generated private key). Furthermore, such a process allows the surgical hub 206 to authenticate the connected component / device itself. Specifically, the supplier / manufacturer (e.g., just authenticated) encrypted the component's serial number using the delivered private key. Upon secure receipt of the private key, the surgical hub 206 can decrypt the encrypted serial number (i.e., read from the connected component), which authenticates the component and / or its association with the supplier / manufacturer (i.e., only the private key received from that supplier / manufacturer will decrypt the encrypted serial number). The surgical hub 206 may still further verify the component as authentic (e.g., by comparing the decrypted serial number to a comprehensive list of authorized serial numbers received separately from the supplier / manufacturer).It should be noted that such aspects described above may alternatively be implemented by the cloud-based system 205 and / or a combination of the cloud-based system 205 and the surgical hub 206 to control which components (e.g., surgical devices / instruments 235, energy devices 241) are utilized in the interactive surgical system 200 (e.g., to perform a surgical procedure) by authenticating the components and / or their source / manufacturer. In one aspect, such described approach may prevent the use of knock-off components in the interactive surgical system 200, ensuring the safety and well-being of surgical patients.

[0287] According to another aspect, an electronic chip of a component (e.g., surgical device / instrument 235, energy device 241) may store (e.g., in memory) data associated with the use of that component (i.e., usage data, e.g., number of uses with a limited use device, number of uses remaining, firing algorithm executed, designation as a single-use component). In one such aspect, upon / after connection of a component to the interactive surgical system, surgical hub 206 and / or cloud-based system 205 may read such usage data from the component's memory and write at least a portion of that usage data back (e.g., individually and / or under the blockchain approach discussed herein) for storage at surgical hub 206 (e.g., in memory 249) and / or for storage at cloud-based system 205. According to one such aspect, surgical hub 206 and / or cloud-based system 205 may again read such usage data upon / after subsequent connection of that component and compare the usage to the previously stored usage data. Here, if a mismatch exists or if predetermined / authorized uses are met, the surgical hub 206 and / or cloud-based system 205 may prevent use of that component (e.g., blacklisted, deactivated, flagged for removal) on the interactive surgical system 200. In various aspects, such an approach prevents bypass of the encryption chip system. If the component's electronic chip / memory is tampered with (e.g., memory reset, changed number of uses, changed firing algorithm, single-use device designated as a multi-use device), a mismatch exists and use of the component is controlled / prevented.

[0288] Further details are disclosed in U.S. Patent No. 9,011,427, entitled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued April 21, 2015, which is incorporated herein by reference in its entirety.

[0289] A surgical hub may be provided that may provide coordination of device pairing in an operating room. One of the functions of the surgical hub 106 is to pair (also referred to herein as "connecting" or "coupling") with other components of the surgical system 102 to control interactions between, gather information from, or coordinate the components of the surgical system 102. Because operating rooms in a hospital are likely to be in close physical proximity to one another, the surgical hub 106 of a surgical system 102 may unknowingly pair with components of the surgical system 102 in an adjacent operating room, which may significantly interfere with the function of the surgical hub 106. For example, the surgical hub 106 may unintentionally activate surgical instruments in a different operating room or record information from a different ongoing surgical procedure in an adjacent operating room.

[0290] Aspects of the present disclosure present a surgical hub 106 that can pair with detected devices of a surgical system 102 located within the boundaries of that operating room. The surgical hub 106 can avoid accidentally pairing with devices in another operating room.

[0291] Additionally, the surgical hub 106 may rely on its knowledge of the location of other components of the surgical system 102 within the operating room when making decisions about, for example, which surgical instruments should be paired with each other or activated. Changes in the location of the surgical hub 106 or other components of the surgical system 102 can be problematic.

[0292] Aspects of the present disclosure further provide a surgical hub 106 that may be configured to reassess or re-determine the boundaries of its operating room upon detecting that the surgical hub 106 has been moved.

[0293] Aspects of the present disclosure further provide a surgical hub 106 that may be configured to re-determine the boundaries of its operating room upon detection of a potential device in the surgical system 102, which may be an indication that the surgical hub 106 has been moved.

[0294] In various aspects, the surgical hub 106 may be used with the surgical system 102 in a surgical procedure performed in an operating room. The surgical hub 106 may include control circuitry configured to determine the boundaries of the operating room, determine the devices of the surgical system 102 located within the boundaries of the operating room, and pair the surgical hub 106 with the devices of the surgical system 102 located within the boundaries of the operating room.

[0295] In one aspect, the control circuitry may be configured to determine the operating room boundaries after power-up of the surgical hub 106. In one aspect, the surgical hub 106 includes communication circuitry configured to detect and pair with devices of the surgical system located within the operating room boundaries. In one aspect, the control circuitry is configured to re-determine the operating room boundaries after potential devices of the surgical system 102 are detected. In one aspect, the control circuitry is configured to periodically determine the operating room boundaries.

[0296] In one aspect, the surgical hub 106 may include an operating room mapping circuit that includes a plurality of non-contact sensors configured to measure the boundaries of the operating room.

[0297] In various aspects, the surgical hub 106 includes a processor and a memory coupled to the processor. The memory stores instructions executable by the processor to pair the surgical hub with devices of the surgical system 102 located within the confines of the operating room, as described above. In various aspects, the present disclosure provides a non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a machine to pair the surgical hub 106 with devices of the surgical system 102 located within the confines of the operating room, as described herein.

[0298] 32 and 33 are logic flow diagrams of a process illustrating the control program or logic configuration for pairing the surgical hub 106 with devices of the surgical system 102 located within the confines of the operating room, as described herein.

[0299] The surgical hub 106 performs a wide range of functions that may use short- and long-range communications, such as assisting in surgical procedures, coordinating among devices in the surgical system 102, and collecting and transmitting data to the cloud 104. To perform its functions, the surgical hub 106 may include a communications module 130 capable of short-range communications with other devices in the surgical system 102. The communications module 130 is also capable of long-range communications with the cloud 104.

[0300] The surgical hub 106 may also include an operating room mapping module 133 that may be capable of identifying the boundaries of the operating room and identifying devices of the surgical system 102 within the operating room. The surgical hub 106 may be configured to identify the boundaries of the operating room and to pair or connect only with potential devices of the surgical system 102 that are detected within the operating room.

[0301] In one aspect, pairing may include establishing a communication link or path, hi another aspect, pairing may include establishing a control link or path.

[0302] Mapping or assessment of the operating room boundaries occurs during startup (e.g., initial startup) of the surgical hub 106. The surgical hub 106 may be configured to maintain spatial awareness during surgery by periodically mapping its operating room, which may be useful in determining if the surgical hub 106 has been moved. Reassessment 3017 may occur periodically or may be triggered by an event, such as observing a change in the devices of the surgical system 102 that are considered to be within the operating room. In one aspect, the change is the detection 3010 of a device (e.g., a new device) that was not previously considered to be within the operating room boundaries, as shown in FIG. 34. In another aspect, the change may be the loss, disconnection, or unpairing of a paired device that was previously considered resident within the operating room, as shown in FIG. 35. The surgical hub 106 may continuously monitor (3035) its connections with paired devices to detect (3034) the loss, disconnection, or unpairing of a paired device.

[0303] In other aspects, the re-evaluation trigger event may be, for example, the detection of a change in the surgeon's position, an instrument change, or a new set of tasks being performed by the surgical hub 106 .

[0304] In one aspect, evaluation of the room boundaries by the surgical hub 106 is achieved by activation of a sensor array of the operating room mapping module 133 within the surgical hub 106, which enables the surgical hub 106 to detect the walls of the operating room.

[0305] Other components of the surgical system 102 may be made spatially aware in the same or similar manner as the surgical hub 106. For example, the robotic hub 122 may also include an operating room mapping module 133. The primary display and / or secondary display may also include an operating room mapping module.

[0306] The surgical hub 106's spatial awareness and its ability to map the operating room for potential components of the surgical system 102 may enable the surgical hub 106 to make autonomous decisions about whether to include or exclude such potential components as part of the surgical system 102, freeing the surgical staff from handling such tasks. Additionally, the surgical hub 106 may be configured to make inferences about the type of surgical procedure to be performed in the operating room based on information collected, for example, before, during, and / or after the performance of the surgical procedure. Examples of collected information include the type of equipment being brought into the operating room, the introduction time of such equipment into the operating room, and / or the device activation sequence. The surgical hub 106's spatial awareness may also be used to update one or more displays in the operating room. For example, the surgical hub 106's spatial awareness may display data on a primary display, display data on a secondary display, and / or move data between the primary and secondary displays based on at least one of instrument detection, operating room mapping, user detection, changes in the surgical hub's location, instrument disconnection, etc.

[0307] In one aspect, the surgical hub 106 uses an operating room mapping module 133 to determine the boundaries of the surgical field (e.g., fixed, mobile, or temporary operating rooms or spaces) using either ultrasound or laser non-contact measurement devices.

[0308] 31, an ultrasound-based non-contact sensor 3002 may be used to scan the surgical field by transmitting bursts of ultrasound and receiving echoes as they reflect off the perimeter walls 3006 of the surgical field to determine the size of the surgical field and adjust Bluetooth pairing distance limits. In one example, the non-contact sensor 3002 may be a ping ultrasonic distance sensor, as shown in FIG.

[0309] FIG. 31 shows how the ultrasonic sensor 3002 transmits a simple chirp using its ultrasonic speaker 3003, allowing the microcontroller 3004 of the operating room mapping module 133 to measure how long it takes for the echo to return to the ultrasonic sensor's ultrasonic microphone 3005. The microcontroller 3004 sends a pulse to the ultrasonic sensor 3002 to begin the measurement. The ultrasonic sensor 3002 then waits long enough for the microcontroller program to initiate a pulse input command. Approximately at the same time that the ultrasonic sensor 3002 chirps a 40 kHz tone, it transmits a high signal to the microcontroller 3004. When the ultrasonic sensor 3002 detects an echo using its ultrasonic microphone 3005, it changes the high signal back to low. The microcontroller's pulse input command measures the time between the high and low transitions and stores the measurement in a variable. This value can be used, along with the speed of sound in air, to calculate the distance between the surgical hub 106 and the operating room wall 3006.

[0310] 31, the surgical hub 106 may include four ultrasonic sensors 3002, each configured to assess the distance between the surgical hub 106 and the walls of the operating room 3000. The surgical hub 106 may include more or less than four ultrasonic sensors 3002 to determine the boundaries of the operating room.

[0311] Other distance sensors may be employed by the operating room mapping module 133 to determine the boundaries of the operating room. In one example, the operating room mapping module 133 may include one or more photoelectric sensors that may be employed to assess the boundaries of the operating room. In one example, a suitable laser distance sensor may also be used to assess the boundaries of the operating room. A laser-based non-contact sensor may scan the operating room by transmitting a laser light pulse, receiving the laser light pulse bouncing off the perimeter walls of the operating room, comparing the phase of the transmitted pulse with the received pulse, determining the size of the operating room, and adjusting the Bluetooth pairing distance limit.

[0312] Referring to the upper left corner of FIG. 47, the surgical hub 106 is brought into the operating room 3000. The surgical hub 106 is activated at the start of setup, which occurs prior to the surgical procedure. In the example of FIG. 47, setup begins at an actual time of 11:31:14 (EST) based on the real-time clock. However, at the setup start time for the described procedure, the surgical hub 106 begins an artificially randomized real-time clock timing scheme at an artificial real-time of 07:36:00 to protect personal patient information (3001).

[0313] At artificial real time 07:36:01, the operating room mapping module 133 uses an ultrasonic distance sensor to ultrasonically ping the operating room (e.g., by sending an ultrasonic burst as described above and waiting for an echo when it reflects off the perimeter walls of the operating room) to verify the size of the operating room and adjust the pairing distance limit.

[0314] At pseudo real time 07:36:03, the data is stripped and time-stamped. At artificial real time 07:36:05, the surgical hub 106 begins pairing devices located only within the operating room 3000 as verified using the ultrasonic distance sensor 3002 of the operating room mapping module 133. The upper right corner of FIG. 33 shows several exemplary devices that are within the boundaries of the operating room 3000 and paired with the surgical hub 106, including a secondary display device 3020, a secondary hub 3021, a common interface device 3022, a powered stapler 3023, a video tower module 3024, and a powered handheld cutting instrument 3025. Meanwhile, secondary hub 3021′, secondary display device 3020′, and powered stapler 3026 are all outside the boundaries of the operating room 3000 and therefore are not paired with the surgical hub 106.

[0315] In addition to establishing communication links with the devices of the surgical system 102 in the operating room, the surgical hub 106 also assigns each of the devices a unique identification and communication sequence or number. The unique sequence may include the name of the device and a timestamp of when communication was first established. Other suitable device information may also be incorporated into the device's unique sequence.

[0316] As shown in the upper left corner of FIG. 47 , the surgical hub 106 has determined that the boundaries of an operating room 3000 are distances a, -a, b, and -b from the surgical hub 106. Because device "D" is outside the determined boundaries of that operating room 3000, the surgical hub 106 will not pair with device "D." FIG. 32 is an exemplary algorithm illustrating how the surgical hub 106 may pair (e.g., only pair) with devices within the boundaries of its operating room. After startup, the surgical hub 106 determines (3007) the boundaries of the operating room using the operating room mapping module 133, as described above. After the initial determination, the surgical hub 106 continuously searches for or detects (3008) devices within a pairing range. If a device is detected (3010), the surgical hub 106 then determines (3011) whether the detected device is within the boundaries of the operating room. If the device is determined to be within the operating room boundaries, the surgical hub 106 pairs with the device (3012). The surgical hub 106 may display data associated with the paired device on the primary and / or secondary display. In certain cases, the surgical hub 106 also assigns an identifier to the device (3013). However, if the surgical hub 106 determines that the detected device is outside the operating room boundaries, the surgical hub 106 will ignore the device (3014).

[0317] 33 , after the initial determination of the operating room boundaries and the initial pairing of devices located within those boundaries, the surgical hub 106 continues to detect (3015) new devices that become available for pairing. If a new device is detected (3016), the surgical hub 106 is configured to re-evaluate (3017) the operating room boundaries before pairing with the new device. If the new device is determined to be within the newly determined operating room boundaries (3018), the surgical hub 106 pairs (3019) with the device and assigns (3030) a unique identifier to the new device. However, if the surgical hub 106 determines that the new device is outside the newly determined operating room boundaries, the surgical hub 106 will ignore (3031) the device.

[0318] For pairing purposes, the operating room mapping module 133 may contain a compass and integrated Bluetooth transceiver. Other communication mechanisms that are not significantly affected by the hospital environment or geographic location may also be used. Bluetooth Low Energy (BLE) beacon technology can currently achieve indoor distance measurements with an accuracy of approximately 1-2 meters, with improved accuracy at closer ranges (within 0-6 meters). To improve the accuracy of distance measurements, a compass is used in conjunction with BLE. The operating room mapping module 133 utilizes BLE and a compass to determine where the module is located relative to the patient. For example, two modules facing each other (detected by the compass) may clearly indicate that the modules are on either side of the patient, with a distance between them exceeding 1 meter. The more "hub"-enabled modules present in the operating room, the greater the achievable accuracy due to triangulation techniques.

[0319] In situations where multiple surgical hubs 106, modules, and / or other peripheral devices are present in the same operating room, as shown in the upper right corner of FIGURE 47, the operating room mapping module 133 is configured to map the physical location of each module present in the operating room. This information may be used by the user interface to display a virtual map of the operating room, allowing the user to more easily identify which modules are present and enabled and their current status. In one aspect, mapping data collected by the surgical hubs 106 is uploaded to the cloud 104, and the data is analyzed to identify, for example, how the operating room is physically set up.

[0320] The surgical hub 106 is configured to determine the device's location by evaluating the strength and direction of transmitted wireless signals. For the Bluetooth protocol, Received Signal Strength Indication (RSSI) is a measurement of the received wireless signal strength. In one aspect, the devices of the surgical system 102 may include a USB Bluetooth dongle. The surgical hub 106 may scan USB Bluetooth beacons to obtain distance information. In another aspect, multiple high-gain antennas on a Bluetooth access point with variable attenuators may produce more accurate results than RSSI measurements. In o...

Claims

1. 1. A surgical hub for controlling a display, said surgical hub comprising: a processor, the processor comprising: determining a user, a medical instrument, and a location of said user and said medical instrument within an operating room; determining contextual data associated with the medical instrument based on the user, the medical instrument, and the location of the user and the medical instrument within the operating room; and transmitting display instructions to the display instructing the display to be configured according to the context data associated with the medical instrument.

2. The surgical hub of claim 1 , wherein the processor is further configured to determine display content related to the contextual data associated with the medical instrument.

3. The surgical hub of claim 2 , wherein the display instructions further include the display content.

4. the processor: determining that the medical instrument is at the location of the medical instrument within the operating room; determining that the location of the user within the operating room is more than a threshold distance from the location of the medical instrument within the operating room; determining that the location of the user and the medical instrument within the operating room indicates that the medical instrument is to be powered off; and setting the display instruction to indicate that the medical instrument should be powered off.

4. The surgical hub of claim 1, wherein the surgical hub is configured to determine the context data associated with the medical instrument based on the user, the medical instrument, and the locations of the user and the medical instrument within the operating room.

5. 5. The surgical hub of claim 1, wherein the display instructions to the display instructing the display to be configured according to the context data associated with the medical instrument cause the display to turn off or remove instrument data.

6. the processor: determining that the location of the medical instrument within the operating room and the location of the user within the operating room are within a threshold distance; determining that the location of the user and the medical instrument within the operating room indicates that the medical instrument is to be cleaned; and setting the display instructions to indicate that the medical instrument should be in a cleaning mode.

6. The surgical hub of claim 1, wherein the surgical hub is configured to determine the contextual data associated with the medical instrument based on the user, the medical instrument, and the location of the user and the medical instrument within the operating room.

7. 7. The surgical hub of claim 1, wherein the display instructions to the display instructing the display to be configured according to the context data associated with the medical instrument cause the display to provide cleaning instructions to the user for the medical instrument.

8. the medical device is a first medical device, and the processor: determining that a location of the first medical instrument within the operating room, a location of a second medical instrument within the operating room, and the location of the user within the operating room are within a threshold distance; determining that the user is replacing the second medical device with the first medical device; and setting the display instructions to indicate that the second medical instrument has been replaced with the first medical instrument. The surgical hub of claim 1, further comprising: a display instruction for displaying the second medical instrument in response to a user request; and a display instruction for displaying the second medical instrument in response to a user request;

9. 9. The surgical hub of claim 8, wherein the display instructions to the display instructing the display to be configured according to the context data associated with the first medical instrument cause the display to add first instrument data associated with the first medical instrument and remove second instrument data associated with the second medical instrument.

10. the context data is first context data, the display instructions are first display instructions, the display is a first display, and the processor: determining second context data associated with the second medical instrument based on the user, a second medical instrument, and a location of the user and the second medical instrument within the operating room; 10. The surgical hub of claim 1, further configured to: turn off a second display; or send a second display instruction to the second display instructing the second display to configure itself according to the second context data by displaying one or more of instructions to reload the second medical instrument, instructions to clean the second medical instrument, or surgical instructions for using the second medical instrument.

11. 10. The surgical hub of claim 1, wherein the display instructions to the display instructing the display to be configured according to the context data of the medical instrument cause the display to show instrument data or instructions for using the medical instrument.

12. the user is a first user, and the processor: determining that the medical device is being moved from a second user to the first user near a patient; and setting the display instructions to indicate that the first user is controlling the medical instrument and that the medical instrument will be used to perform a surgical procedure task, thereby determining the context data for the medical instrument based on the first user, the medical instrument, and their locations within the operating room.

13. 13. The surgical hub of claim 1, wherein the context data indicates that the user is controlling the medical instrument, and the display instructions include instructions to cause the display to show one or more of instrument data, medical instrument instructions, and surgical procedure instructions.

14. 14. The surgical hub of claim 1, wherein the context data indicates that the user is controlling the medical instrument, and the display instructions include instructions to cause the display to show instrument data based on one or more of an orientation of the medical instrument, a handedness of the user, and a level of inversion of the medical instrument.

15. 15. The surgical hub of any one of claims 1-14, wherein the processor is configured to determine the user, the medical instrument, and the location of the user and the medical instrument within the operating room using one or more of a camera, a sensor in the operating room, a sensor associated with the user, a sensor associated with the medical instrument, and a wearable device.

16. The surgical hub of any one of claims 1 to 15, wherein the user is one or more of a patient, a healthcare provider, a doctor, a nurse, a delivery nurse, and a medical technician.

17. The surgical hub of any one of claims 1 to 16, wherein the display is a primary display or a secondary display.

18. 1. A method for a surgical hub to control a display, the method comprising: determining a user, a medical instrument, and a location of said user and said medical instrument within an operating room; determining contextual data associated with the medical instrument based on the user, the medical instrument, and the location of the user and the medical instrument within the operating room; and transmitting display instructions to the display instructing the display to be configured according to context data associated with the medical device.

19. The method of claim 18 , further comprising determining display content related to the contextual data associated with the medical device.

20. The method of claim 19 , wherein the display instructions further include the display content.

Images