Communication control for secondary and primary displays in surgical control systems.

Abstract

The layered multi-display control scheme may provide various communication control options for the surgeon-controlled secondary display and the primary operating room display. Powered surgical tools may be in operative communication with a local display and at least one primary monitor in the operating room outside the sterile field to display multiple data and / or imaging sources. The local display may be interactable by the surgeon within the sterile field. A display outside the sterile field may show an image in the form of a laparoscopic scope and may include other data streams superimposed from other devices other than the scope. The secondary display may be used to direct its displayed content onto or remove it from the primary display. The added or removed data stream may originate from, pass through, or be networked with the secondary display.

Description

Technical Field

[0001] (Cross - reference to related applications) This application is related to the following applications filed simultaneously, the content of each of which is incorporated herein by reference. · Attorney docket number END9287USNP1, titled "METHOD FOR OPERATING TIERED OPERATION MODES IN A SURGICAL SYSTEM".

Background Art

[0002] Surgical systems often incorporate an imaging system that allows a clinician to view the surgical site and / or one or more portions thereof on one or more displays, such as a monitor. The display may be local to the surgical theater and / or remote. The imaging system can include a scope with a camera that views the surgical site and transmits a view to a display that the clinician can view. Scopes can include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophago - duodenoscopes (gastric cameras), endoscopes, laryngoscopes, nasopharyngo - pyeloscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and external scopes. The imaging system may be limited by the information it can recognize and / or communicate to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three - dimensional space may not be recognizable during surgery with a particular imaging system. Additionally, certain imaging systems may not be able to communicate and / or transmit certain information to the clinician during surgery.

Summary of the Invention

Means for Solving the Problems

[0003] According to various embodiments of the present invention, the following examples are provided.

[0004] 1. A surgical instrument supplied with power, A communication array operably connected to, for example, a first display which may be located inside the surgical sterile field, and a second display which may be located outside the surgical sterile field, A processor, and the processor, To obtain multi-display control parameters, Identifying the current multi-display control mode based on multi-display control parameters, Based on the current multi-display control mode, determine whether to generate visualization control data associated with the second display, A powered surgical instrument configured to interact with a first display and a second display based on that determination.

[0005] 2. A powered surgical instrument as described in Example 1, wherein a communication array is operably connected to a surgical hub, and multi-display control parameters include instructions from the surgical hub.

[0006] 3. A powered surgical instrument according to Embodiment 1 or 2, wherein a communication array is operably connected to a surgical hub, and the processor is further configured to receive content from the surgical hub for display on a first display and to transmit the received content to the first display.

[0007] 4. A powered surgical instrument according to any one of Examples 1 to 3, wherein the processor is further configured to acquire visualization control data associated with a second display via a first display, based on a determination that the current multi-display control mode supports sterile field display-based control, and to transmit the visualization control data associated with the second display to a surgical hub in order to control the second display.

[0008] 5. A powered surgical instrument according to any one of Examples 1 to 4, wherein the processor is further configured to disable the generation of visualization control data associated with a second display based on a determination that the current multi-display control mode does not support sterile field display-based control.

[0009] 6. A powered surgical instrument according to any one of Examples 1 to 5, wherein the processor is configured to receive a user instruction to change the content on a second display based on a determination that the current multi-display control mode supports sterile field display-based control; generate visualization control data associated with the second display based on the received user instruction; and transmit the visualization control data associated with the second display to the surgical hub in order to control the second display.

[0010] 7. The powered surgical instrument according to Embodiment 6, wherein a user instruction to change the content on the second display is received via the first display.

[0011] 8. A powered surgical instrument according to Example 6 or 7, wherein a user instruction indicates at least one of projecting content associated with a first display onto a second display, or removing content associated with a first display from a second display.

[0012] 9. A powered surgical instrument is equipped with at least one sensor for sensing surgical data, and the processor determines whether to request aggregated analysis from a remote server via the surgical hub, based on the current multi-display control mode. Based on the determination that aggregate analysis is required, the process involves generating an aggregate analysis request and receiving an aggregate analysis response via a communication array. A powered surgical instrument from any one of Examples 1 to 8, further configured to combine the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on a first display.

[0013] 10. A powered surgical instrument according to any one of Examples 1 to 9, wherein the powered surgical instrument comprises a first display inside the surgical sterile field.

[0014] 11. A powered surgical instrument as described in any one of Examples 1 to 10, wherein the first display inside the surgical sterile field is located outside the powered surgical instrument.

[0015] In all of the embodiments described above, where a first display is mentioned, it is preferably a first display inside the surgical sterile field. In all of the embodiments described above, where a second display is mentioned, it is preferably a second display outside the surgical sterile field.

[0016] 12. A surgical hub comprising a communication array operably connected to a first display, which may be located inside a surgical sterile field, and a second display, which may be located outside a surgical sterile field, A surgical hub comprising a processor, the processor configured to acquire multi-display control parameters, identify the current multi-display control mode based on the multi-display control parameters, determine whether to receive visualization control data associated with the second display from the first display based on the current multi-display control mode, and interact with the first and second displays based on that determination.

[0017] 13. The processor, The surgical hub according to Example 12, further configured to receive visualization control data associated with a second display based on a determination that the current multi-display control mode supports sterile field display-based control, and to control the second display based on the received visualization control data associated with the second display.

[0018] 14. The processor, Based on the current multi-display control mode, determine whether to retrieve the aggregated analysis from the remote server for display on the first display, Based on the determination that the current multi-display control mode supports remote aggregation, an aggregation analysis request is generated, Receiving aggregated analysis responses via a communication array, The surgical hub according to Example 12 or 13, further configured to combine the received aggregated analysis response with surgical data received from at least one instrument in order to generate content to be displayed on a first display.

[0019] 15. A method for communicating with a first display inside a surgical sterile field and a second display outside a surgical sterile field, comprising: obtaining multi-display control parameters; identifying a current multi-display control mode based on the multi-display control parameters; determining whether to generate visualization control data associated with a second display outside a surgical sterile field based on the current multi-display control mode; and interacting with the first display inside a surgical sterile field and the second display outside a surgical sterile field based on the determination.

[0020] 16. Based on the determination that the current multi-display control mode supports the control of the sterile field display base, obtaining visualization control data associated with a second display outside the surgical sterile field via a first display inside the surgical sterile field; and transmitting the visualization control data associated with the second display to control the second display outside the surgical sterile field. The method according to embodiment 15 further includes these steps.

[0021] 17. Based on the determination that the current multi-display control mode does not support the control of the sterile field display base, further including invalidating the generation of visualization control data associated with a second display outside the surgical sterile field. The method according to embodiment 15 includes this step.

[0022] 18. Based on the determination that the current multi-display control mode supports the control of the sterile field display base, receiving a user instruction to change the content on a second display outside the surgical sterile field; generating visualization control data associated with a second display outside the surgical sterile field based on the received user instruction; and transmitting the visualization control data associated with a second display outside the surgical sterile field to control the second display outside the surgical sterile field. The method according to embodiment 15 further includes these steps.

[0023] 19. The method according to embodiment 18, wherein a user instruction to change the content on a second display outside the surgical sterile field is received via a first display inside the surgical sterile field.

[0024] 20. Based on the current multi-display control mode, determining whether to request an aggregated analysis from a remote server; generating an aggregated analysis request based on the determination to request an aggregated analysis; receiving an aggregated analysis response; Receiving detected surgical data from at least one sensor, The method according to Example 15, further comprising combining the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on a first display inside the surgical sterile field.

[0025] In the embodiments described above, including at least Embodiment 1, a powered surgical tool may be configured to selectively interact with a first display and a second display. For example, if the first display is inside the surgical sterile field and the second display is outside the surgical sterile field, this allows the powered surgical tool to selectively interact with displays in different parts of the operating room, which may be in the same room or different rooms. To ensure strict adherence to medical procedures concerning the sterile field, medical professionals may benefit from utilizing multiple displays located inside and outside the sterile field. The tool is configured such that selective interaction is controlled by multi-display control parameters. In embodiments, the multi-display control parameters may include one or more instructions, such as instructions from a surgical hub, consumer control parameters such as subscription levels, or instructions from a hierarchical system. In other embodiments, the multi-display control parameters may include software or hardware parameters such as available data bandwidth, power capacity and usage, and processor and memory utilization. These software or hardware parameters may belong to a powered surgical tool, a surgical hub, a first and / or second display, or any system between them.

[0026] Powered surgical devices can operate under various multi-display control modes, such as one-way communication mode, sterile field display-based control mode, and / or remote aggregation analysis mode. In at least one embodiment (3), the tool is configured to operate under one-way communication mode. When operating in an exemplary one-way communication mode, the surgical device can receive and transmit content to a display inside the surgical sterile field. Content to be displayed on the display inside the surgical sterile field may be received from a surgical hub. This can be useful for medical professionals, such as surgeons or nurses, within the sterile field to view information about the procedure being performed. In at least one embodiment (4) and one embodiment (6), the tool is configured to operate under sterile field display-based control mode. Embodiment 9 corresponds to remote aggregation analysis mode.

[0027] Each mode or hierarchy enables different levels of communication control tailored to the requirements associated with the mode of each hierarchy or display. Adaptive control allows surgeons to be provided with appropriate information on the display. For example, in the basic mode or hierarchy, the secondary display simply serves as a place where surgeons can view or access a portion of the overall data environment. More sophisticated modes or hierarchies allow the secondary display to function not only as another display but also as a control system, thereby enabling control over systems within the connected environment, the location of information display, and the location of information storage. The secondary display may further be a display portal for surgeons to access aggregated or compiled data on a remote server, where historical datasets, usage processes, problem solving, images or videos of value for the current procedure, comparative information from previous patients existing in a similar format, or procedure suggestions may reside. In this most complex mode or hierarchy of operation, the secondary display would communicate through data and analysis residing on a remote server within the local display, in combination with locally created content and data, enabling their access and display.

[0028] The various multi-display control modes described above provide hierarchical multi-display control with various communication control options for secondary and primary operating room displays controlled by medical professionals. Designed for use within a sterile field and accessible for input and display by the surgeon, the display unit provides the surgeon with interactive input control from the sterile field to control other surgical displays coupled to a surgical hub. A secondary user interface via the display unit may enable control of non-sterile displays from within the sterile field and may provide suitable visualization control. In the above embodiments, the level or hierarchy may be controlled or limited by hardware, the communication capabilities of the systems involved, or user input. Powered surgical instruments can adaptively control interaction with secondary displays inside the surgical sterile field and primary displays outside the surgical sterile field for hierarchical control. Hierarchical multi-display control allows powered surgical instruments and a surgical hub to adaptively control interaction with secondary displays in the sterile field. Through interaction with secondary and primary displays, the surgeon may be provided with a wide range of information and enhanced control. This can further improve the quality and safety of the operation.

[0029] The methods of Examples 15-20 and the surgical hubs of Examples 12-14 correspond to the devices of Examples 1-11. Therefore, the above considerations regarding Examples 1-1 also apply to Examples 12-20.

[0030] According to one embodiment of the present invention, a powered surgical device may include a processor configured to acquire one or more multi-display control parameters and to identify the current multi-display control mode based on those multi-display control parameters. Based on the current multi-display control mode, it may be determined whether to generate visualization control data associated with the display outside the surgical sterile field. Based on that determination, the powered surgical device may interact with the display inside the surgical sterile field and the display outside the surgical sterile field.

[0031] Multi-display control parameters may include instructions from the surgical hub. Multi-display control parameters may include consumer control parameters such as subscription levels. Multi-display control parameters may include available data bandwidth, power capacity and usage, processor and memory utilization, and / or internal or auxiliary systems. Multi-display control parameters may include instructions from hierarchical systems.

[0032] Powered surgical devices can operate under various multi-display control modes, such as one-way communication mode, sterile field display-based control mode, and / or remote aggregated analysis mode.

[0033] For example, when operating in an exemplary one-way communication mode, a surgical device may receive content for display on a display inside the surgical sterile field and transmit the received content to the display. Content for display on the display inside the surgical sterile field may be received from a surgical hub.

[0034] For example, when operating in an exemplary sterile field display-based control mode, a surgical instrument may obtain visualization control data associated with a display outside the surgical sterile field via a display inside the surgical sterile field. The surgical instrument may transmit visualization control data associated with the display (e.g., to a surgical hub) to control the display outside the surgical sterile field. When operating under a multi-display control mode does not support sterile field display-based control, the surgical instrument may disable the generation of visualization control data associated with the display outside the surgical sterile field.

[0035] A surgical instrument may determine that the current multi-display control mode supports sterile field display-based control and may receive a user instruction to change the content on a display outside the surgical sterile field. Based on the received user instruction, the surgical instrument may generate visualization control data associated with the display outside the surgical sterile field and transmit this visualization control data (e.g., to the surgical hub) to control the display outside the surgical sterile field. A user instruction to change the content on a display outside the surgical sterile field may be received via a display inside the surgical sterile field. The user instruction may instruct to project the content associated with the display inside the surgical sterile field onto the display outside the surgical sterile field, or to remove the content associated with the display inside the surgical sterile field from the display outside the surgical sterile field.

[0036] For example, when operating in an exemplary remote aggregate analysis mode, a surgical instrument may request aggregate analysis from a remote server (e.g., via a surgical hub). Based on the current multi-display control mode, the surgical instrument may determine whether to request aggregate analysis from the remote server (e.g., via a surgical hub). Based on the determination to request aggregate analysis, an aggregate analysis request may be generated. The aggregate analysis response is received and may be combined with surgical data generated based on the sensed surgical data for display on a display inside the surgical sterile field.

[0037] The surgical hub may include a communication array operably connected to surgical instruments, displays inside the surgical sterile field, and displays outside the surgical sterile field. The surgical hub may include a processor configured to acquire multi-display control parameters and identify the current multi-display control mode based on the multi-display control parameters. Based on the current multi-display control mode, the surgical hub may interact with at least one display inside the surgical sterile field and at least one display outside the surgical sterile field. For example, based on the current multi-display control mode, the surgical hub may determine whether to receive visualization control data associated with the displays outside the surgical sterile field from the displays inside the surgical sterile field.

[0038] The surgical hub can operate under various multi-display control modes, such as one-way communication mode, sterile field display-based control mode, and / or remote aggregation analysis mode.

[0039] Based on the determination that the current multi-display control mode supports sterile field display-based control, the surgical hub may receive visualization control data associated with displays outside the surgical sterile field and control those displays based on the received visualization control data associated with those displays.

[0040] The surgical hub may determine, based on the current multi-display control mode, whether to retrieve aggregated analysis from a remote server for display on a first display inside the surgical sterile field. Based on the determination that the current multi-display control mode supports remote aggregation, the surgical hub may generate an aggregated analysis request. The aggregated analysis request may be generated based on instructions from a display inside the sterile field. A surgical instrument may receive an aggregated analysis response and combine the received aggregated analysis response with surgical data received from at least one instrument to generate content for display on the first display inside the surgical sterile field. [Brief explanation of the drawing]

[0041] [Figure 1] This is a block diagram of a computer-implemented interactive surgical system. [Figure 2] This shows an exemplary surgical system used for performing surgical procedures in the operating room. [Figure 3] This disclosure illustrates at least one aspect of a visualization system, a robotic system, and a surgical hub paired with an intelligent instrument. [Figure 4] An example of a surgical data network provided by at least one aspect of this disclosure is a communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility specially equipped for surgical procedures, to the cloud. [Figure 5] This example illustrates a computer-implemented interactive surgical system. [Figure 6] An exemplary surgical hub is shown, comprising multiple modules coupled to a modular control tower. [Figure 7] An example of a surgical instrument or tool is shown. [Figure 8] An exemplary surgical instrument or tool having a motor that can be activated to perform various functions is illustrated. [Figure 9]This is a diagram illustrating an exemplary situational awareness surgical system. [Figure 10] This document illustrates an exemplary timeline of a surgical procedure and the inferences that a surgical hub can make from the data detected at each step of the surgical procedure. [Figure 11] This is a block diagram of a computer-implemented interactive surgical system. [Figure 12] This document illustrates the functional architecture of an exemplary computer-implemented interactive surgical system. [Figure 13] An exemplary computer-implemented interactive surgical system is provided, configured to adaptively generate control program updates for modular devices. [Figure 14] An exemplary surgical system is provided, which includes a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter. [Figure 15A] This section describes how to determine the operating mode and provides an example flow for operating in that determined mode. [Figure 15B] An illustrative flow for changing the operating mode is provided. [Figure 16] An example of a primary display in a surgical hub equipped with global and local displays is provided. [Figure 17] An example of a primary display for a surgical hub is provided. [Figure 18] A perspective view is shown illustrating a surgeon wearing a pair of safety glasses while using surgical instruments, including a handle assembly housing and a wireless circuit board, during a surgical procedure. [Figure 19] This shows an exemplary flow for operation under multi-display control mode. [Figure 20] This shows an exemplary flow for operation under hierarchical multi-display control mode. [Figure 21] This shows an exemplary flow for operation under hierarchical multi-display control mode. [Figure 22]This shows exemplary multi-display control modes, such as one-way communication mode. [Figure 23] An exemplary multi-display control mode is shown, which may support sterile field display-based control. [Figure 24] This demonstrates an exemplary multi-display control mode that can support remote data aggregation and analysis. [Figure 25] This shows an exemplary flow for operation under hierarchical multi-display control mode. [Figure 26] This shows an exemplary flow for operation under hierarchical multi-display control mode. [Modes for carrying out the invention]

[0042] The applicant of this application also owns the following concurrently filed U.S. patent applications, each of which is incorporated herein by reference in its entirety. U.S. Patent Application No. 15 / 940,671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER" (Agent Reference Number END8502USNP), filed on March 29, 2018. • U.S. Patent No. 9,011,427, entitled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued on April 21, 2015, and U.S. Patent Application No. 15 / 940,668, titled "AGGREGATION AND REPORTING OF SURGICAL HUB DATA" (Agent Reference Number END8501USNP2), filed on March 29, 2018.

[0043] Referring to Figure 1, the computer-implemented interactive surgical system 100 may include one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 which may include a remote server 113 coupled to a storage device 105). Each surgical system 102 may include at least one surgical hub 106 that communicates with the cloud 104 which may include the remote server 113. In one embodiment, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some embodiments, the surgical system 102 may include M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P are integers of 1 or more.

[0044] In various embodiments, the visualization system 108 may include one or more imaging sensors strategically positioned relative to a sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described under the heading “Advanced Imaging Acquisition Module” in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” which is incorporated herein by reference in its entirety.

[0045] As shown in Figure 2, the primary display 119 is positioned in the sterile field so that it is visible to the operator on the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may include a first non-sterile display 107 and a second non-sterile display 109, facing away from each other. The visualization system 108, guided by the hub 106, is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 can cause the visualization system 108 to display snapshots of the surgical site recorded by the imaging device 124 on the non-sterile displays 107 or 109 while maintaining live video of the surgical site on the primary display 119. The snapshots on the non-sterile displays 107 or 109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0046] In one embodiment, the hub 106 may also be configured to transfer diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a primary display 119 in the sterile field, allowing the sterile operator to view the diagnostic input or feedback on the operating table. In one embodiment, the input may take the form of modifications to a snapshot displayed on the non-sterile display 107 or 109, and the form of modifications may be transferred to the primary display 119 by the hub 106.

[0047] Referring to Figure 2, the surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 may also be configured to coordinate the flow of information to the display of the surgical instrument 112. For example, in U.S. Patent Application Publication 2018-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed December 4, 2019, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 may be transmitted by the hub 106 to the surgical instrument display 115 in the sterile field, allowing the operator of the surgical instrument 112 to view the diagnostic input or feedback. Exemplary surgical instruments suitable for use with surgical system 102 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0048] Figure 2 illustrates an embodiment of a surgical system 102 used to perform surgical procedures on a patient lying on an operating table 114 in a surgical operating room 116. A robotic system 110 may be used as part of the surgical system 102 in a surgical procedure. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. The patient-side cart 120 allows the surgeon to manipulate at least one detachably coupled surgical tool 117 through a minimally invasive incision in the patient's body while viewing the surgical site through the surgeon's console 118. Images of the surgical site can be acquired by a medical imaging device 124, which can be manipulated by the patient-side cart 120 to orient the imaging device 124. Images of the surgical site can be processed using the robotic hub 122 for subsequent display to the surgeon through the surgeon's console 118.

[0049] Other types of robotic systems can be readily adapted for use with surgical system 102. Various embodiments of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed on 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0050] Various examples of cloud-based analytical methods implemented by Cloud 104 and suitable for use with the present disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed on 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0051] In various embodiments, the imaging device 124 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0052] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0053] One or more illumination sources may be configured to emit electromagnetic energy in the invisible spectrum as well as the visible spectrum. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0054] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0055] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, gastroscopy (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0056] The imaging device may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments that sense specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the human eye's receptors for red, green, and blue. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for moving the surgical field after the surgical task is completed in order to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required during any surgical procedure. The strict hygiene and sterilization conditions required in the “surgical field,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient when the surgical procedure is ready. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and fixation devices within that area.

[0057] Referring here to Figure 3, a hub 106 is shown that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, the application of energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Valuable time can be lost dealing with this problem during surgical procedures. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The modular enclosure 136 of the hub provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from a remote surgical site to a suction and irrigation module slidably received within a hub enclosure. In one embodiment, the hub enclosure comprises a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house different generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. A modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy different from the first energy for application to tissue, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and second data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented with respect to a modular enclosure 136 of a hub that enables modular integration of a generator module 140, a smoke evacuation module 126, and a suction / irrigation module 128. The modular enclosure 136 of the hub further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the modular enclosure 136 of the hub. The modular enclosure 136 of the hub may be configured to facilitate the insertion of multiple generators, such that multiple generators function as a single generator, and interactive communication between generators docked within the modular enclosure 136 of the hub.

[0058] Figure 4 illustrates a surgical data network 201 comprising a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility specially equipped for surgical procedures, to a cloud-based system (a cloud 204 which may include, for example, a remote server 213 coupled to a storage device 205). In one embodiment, the modular communication hub 203 comprises a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be coupled to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to travel from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional mechanisms that constitute each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0059] Modular devices 1a-1n located in the surgical field may be coupled to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be coupled to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m located in the same surgical field may also be coupled to a network switch 209. The network switch 209 may be coupled to a network hub 207 and / or a network router 211 to connect devices 2a-2m to the cloud 204. Data associated with devices 2a-2n may be transferred to the cloud 204 via the network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transferred to a local computer system 210 for local data processing and manipulation.

[0060] It will be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 coupled to an endoscope, a generator module 140 coupled to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device coupled to a display, and / or a non-contact sensor module.

[0061] In one embodiment, the surgical data network 201 may include a combination of a network hub, network switches, and network routers connecting devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m coupled to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” can be used as a metaphor for “Internet,” but the term is not limited in that way. Thus, the term “cloud computing” can be used herein to refer to “a type of internet-based computing,” in which various services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 203 and / or computer system 210 located in a surgical setting (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 203 and / or computer system 210. The cloud infrastructure may be maintained by a cloud service provider. In this context, a cloud service provider may be an entity that coordinates the use and control of one or more devices 1a-1n / 2a-2m located within the surgical field. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located within the surgical field. The hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0062] By applying cloud computing data processing techniques to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures and to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. This may include tissue localization and margin confirmation, as well as phenotypic analysis. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204, a local computer system 210, or both for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve surgical outcomes by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics, can be performed on tissue-specific sites and conditions. Such data analysis may also involve prognostic analysis, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0063] The surgical field devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same surgical field network. The network hub 207 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 207 does not need to store any media access control / Internet Protocol (MAC / IP) for transferring device data. Only one of devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 does not need to have a routing table or intelligence regarding where to send information and broadcasts all network data across each connection to the remote server 213 (Figure 4) via the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and could cause bottlenecks.

[0064] Surgical site devices 2a-2m may be connected to network switch 209 via a wired or wireless channel. Network switch 209 operates within the data link layer of the OSI model. Network switch 209 may also be a multicast device for connecting devices 2a-2m located within the same surgical site to the network. Network switch 209 can transmit data in frame form to network router 211 and operates in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0065] The network hub 207 and / or network switch 209 may be coupled to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates a route for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by any one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, such as different networks located in different operating rooms of the same medical facility or different operating rooms of different medical facilities. The network router 211 can send data to the cloud 204 in the form of packets and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to forward data.

[0066] In one embodiment, the network hub 207 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located within the surgical field.

[0067] In this embodiment, surgical site devices 1a-1n / 2a-2m can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with a modular communication hub 203 via the Bluetooth wireless technology standard to build a personal area network (PAN). The surgical site devices 1a-1n / 2a-2m may communicate with the modular communication hub 203 via several wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, new radio (new radio, NR), long-term evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G and beyond. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0068] The modular communication hub 203 can act as a central connection point for one or all of the surgical site devices 1a-1n / 2a-2m and can handle a data type known as a frame. Frames can carry data generated by devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, it is amplified and transmitted to the network router 211, which then transfers the data to cloud computing resources using several wireless or wired communication standards or protocols as described herein.

[0069] The modular communication hub 203 may be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it can be a good choice for networking surgical site devices 1a-1n / 2a-2m.

[0070] Figure 5 illustrates a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to the surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204, which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to multiple surgical site devices, such as intelligent surgical instruments, robots, and other computerized devices located within the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 coupled to a computer system 210.

[0071] As shown in the embodiment of Figure 5, the modular control tower 236 may be coupled to an imaging module 238 coupled to an endoscope 239, a generator module 240 coupled to an energy device 241, a fume exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, and optionally smart devices / instruments 235 coupled to a display 237 and a non-contact sensor module 242. Surgical field devices may be coupled to cloud computing resources and data storage via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the devices / instruments 235 and the visualization system 208 may be coupled to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be coupled to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with images and superimposed images.

[0072] Figure 6 illustrates a surgical hub 206 comprising multiple modules coupled to a modular control tower 236. The modular control tower 236 may comprise a modular communication hub 203, such as a network connectivity device, and a computer system 210, for example, local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 can expand the number of modules (e.g., devices) that can be connected to the modular communication hub 203 and can be connected in a tiered configuration to transfer data associated with the modules to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and a local display 217. Communication to the cloud 204 can be done via either a wired communication channel or a wireless communication channel.

[0073] The surgical hub 206 may have a non-contact sensor module 242 to measure the dimensions of the surgical field and generate a map of the operating room using either an ultrasonic non-contact measuring device or a laser non-contact measuring device. The ultrasonic-based non-contact sensor module may scan the surgical field by transmitting bursts of ultrasound and receiving echoes as they bounce off the surrounding walls of the surgical field, as described under the heading “Surgical Hub Spatial Awareness Within an Operating Room” in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” which is incorporated in its entirety by reference. The sensor module is configured to determine the size of the surgical field and adjust the Bluetooth pairing distance limits. A laser-based non-contact sensor module can scan a surgical site, for example, by transmitting laser light pulses, receiving laser light pulses reflected from the outer wall of the surgical site, comparing the phase of the transmitted pulses with the received pulses to determine the size of the surgical site, and adjusting the Bluetooth pairing distance limit.

[0074] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be coupled via a system bus to a communication module 247, storage 248, memory 249, non-volatile memory 250, and input / output interface 251. The system bus may be any of several types of bus structures, including a memory bus or memory controller, peripheral bus or external bus, and / or local bus, using any various available bus architectures, including but not limited to 9-bit buses, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0075] Processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0076] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0077] System memory can include volatile and non-volatile memory. The basic input / output system (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory can include random-access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0078] The computer system 210 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage devices. Disk storage devices may include, but are not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Other storage media may include, but are not limited to, optical disk drives such as compact disc ROM devices (CD-ROM), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROM). Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

[0079] It should be understood that the computer system 210 may include software that acts as an intermediary between the user and the basic computer resources described in a preferred operating environment. Such software may include an operating system. An operating system that can be stored on disk storage may function to control and allocate the resources of the computer system. System applications can leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0080] The user can input commands or information to the computer system 210 via input devices coupled to the I / O interface 251. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to the output device. Output adapters may be provided to illustrate some output devices that may require special adapters, such as monitors, displays, speakers, and printers. Examples of output adapters include, but are not limited to, video and sound cards that provide means of connection between the output device and the system bus. Note that other devices and / or systems of devices, such as remote computers, may provide both input and output functions.

[0081] Computer system 210 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface and then physically connected via a communication connection. Network interfaces may include communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated services digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0082] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5 and 6, and / or the visualization system 208, and / or the processor module 232 may comprise an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction, multiple data (SIMD) or multiple-instruction, multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may also be a system on a chip with a multi-core processor architecture.

[0083] The communication connection section may refer to hardware / software used to connect a network interface to a bus. For the sake of clarity of the example, the communication connection section is shown as being inside the computer system, but it may also be outside the computer system 210. Hardware / software required for connection to the network interface may include, for illustrative purposes only, internal and external technologies such as modems including typical telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0084] Figure 7 illustrates a logic diagram of a control system 470 for a surgical instrument or surgical tool according to one or more embodiments of the present disclosure. The system 470 may include a control circuit. The control circuit may include a microcontroller 461 having a processor 462 and memory 468. For example, one or more of sensors 472, 474, and 476 provide real-time feedback to the processor 462. A motor 482 driven by a motor driver 492 drives an I-beam knife element by operably coupling a longitudinally movable displacement member. A tracking system 480 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to the processor 462 and can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control the firing of the I-beam, the movement of the occluder, the rotation of the shaft, and the joint movement. The display 473 can display various operating states of the instrument and may include a touch screen function for data input. The information displayed on the display 473 can be overlaid with images acquired via the endoscopic imaging module.

[0085] In one embodiment, the microcontroller 461 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 461 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels.

[0086] In one embodiment, the microcontroller 461 may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0087] The microcontroller 461 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 461 may include a processor 462 and memory 468. The electric motor 482 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 492 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 480 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication No. 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety.

[0088] The microcontroller 461 may be programmed to provide precise control over the velocity and position of the displacement member and joint motion system. The microcontroller 461 may be configured to calculate the response within its software. The calculated response is compared with the measured response of the actual system to obtain an "observed" response, which can be used to determine the actual feedback. The observed response may be a well-adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and this can detect external influences on the system.

[0089] In some embodiments, the motor 482 may be controlled by a motor driver 492 and can be used in a launching system for surgical instruments or surgical tools. In various forms, the motor 482 may be, for example, a brushed DC-driven motor having a maximum rotational speed of about 25,000 RPM. In some embodiments, the motor 482 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 492 may comprise, for example, an H-bridge driver including a field-effect transistor (FET). The motor 482 may be powered by a power supply assembly removably mounted on a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for powering a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one embodiment, the battery cells may be a lithium-ion battery that can be coupled to and detached from the power supply assembly.

[0090] The motor driver 492 may be the A3941, available from Allegro Microsystems, Inc. The A3941 492 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 492 may have a built-in charge pump regulator that can provide full (>10V) gate drive for battery voltages up to 7V and allows the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be easily substituted for use in the tracking system 480 with an absolute positioning system.

[0091] The tracking system 480 may comprise a controlled motor drive circuit arrangement comprising a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for the absolute positioning system can provide a unique position signal corresponding to the location of the displaced member. In some embodiments, the displaced member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some embodiments, the displaced member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some embodiments, the displaced member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displaced member may be used to refer to any movable member of a surgical instrument or surgical tool, such as a drive member, launch member, launch bar, I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be coupled to any position sensor 472 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be coupled to any suitable linear displacement sensor. The linear displacement sensor may include contact-type or non-contact-type displacement sensors.A linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0092] The electric motor 482 may include a rotary shaft that operably interfaces with a gear assembly mounted on a displacement member by meshing with a set of drive teeth or a rack. The sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor 472 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor arrangement can be connected to a linear actuator by a rack and pinion arrangement, or to a rotary actuator by a spur gear or other connection. A power supply can provide power to the absolute positioning system, and an output indicator can display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing with the corresponding drive gear of the gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0093] One rotation of the sensor element associated with the position sensor 472 may be equivalent to the longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element coupled to the displacement member. The sensor arrangement may be coupled via a gear reduction that results in the position sensor 472 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 472 can complete multiple rotations relative to the full stroke of the displacement member.

[0094] To provide a unique position signal for two or more rotations of the position sensor 472, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches may be fed back to a microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 in the sensor arrangement may include an array of analog rotation sensors such as a magnetic sensor, a potentiometer, or an analog Hall effect element, which outputs a unique combination of position signals or values.

[0095] The position sensor 472 may comprise any number of magnetic sensing elements, such as magnetic sensors, which are classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors can encompass many aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0096] In one embodiment, the position sensor 472 of the tracking system 480, which includes an absolute positioning system, may include a magnetic rotation absolute positioning system. The position sensor 472 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor, available from Austria Microsystems, AG. The position sensor 472 interfaces with a microcontroller 461 to provide an absolute positioning system. The position sensor 472 may be a low-voltage, low-power component and may include four Hall effect elements within the area of ​​the position sensor 472, which may be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and Boulder algorithm, may be provided to implement a simple and efficient algorithm for computing hyperbolic and trigonometric functions that requires only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 461 via a standard serial communication interface, such as a serial peripheral interface (SPI). The position sensor 472 can provide 12-bit or 14-bit resolution. The position sensor 472 may also be an AS5055 chip, which is available in a small QFN 16-pin 4x4x0.85mm package.

[0097] The tracking system 480, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is coupled to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. The calculated response of a physical system can take into account properties such as mass, inertia, viscous friction, and inductive resistance to predict what the state of the physical system and its output will be by knowing the input.

[0098] The absolute positioning system can provide the absolute position of a displacement member when the device is powered on, without moving the displacement member back or forward to a reset (zero or home) position, as may be required with a conventional rotary encoder that simply counts the number of forward or backward strokes taken by the motor 482 to estimate the position of a device actuator, drive bar, knife, etc.

[0099] Sensor 474, for example, a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of the end effector, such as the amplitude of strain exerted on the anvil during clamping, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to processor 462. Instead of, or in addition to, sensor 474, sensor 476, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, sensor 476, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and deform into contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 478 may be used to measure the current drawn in by motor 482. The force required to propel the launching member forward may, for example, correspond to the current drawn in by the motor 482. The measured force may be converted into a digital signal and provided to the processor 462.

[0100] In one embodiment, a strain gauge sensor 474 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied to the tissue being treated by the end effector. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 474, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 474 can measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain may be converted into a digital signal and provided to the processor 462 of the microcontroller 461. A load sensor 476 can measure the force used to operate a knife element, for example, to cut tissue trapped between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor may also be converted into a digital signal and provided to the processor 462.

[0101] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 474 and 476 respectively, can be used by the microcontroller 461 to characterize the selected position of the launcher and / or the corresponding values ​​of the launcher's velocity. In one example, memory 468 can store techniques, equations, and / or lookup tables that may be used by the microcontroller 461 during evaluation.

[0102] The control system 470 for surgical instruments or surgical tools may also include a wired communication circuit or a wireless communication circuit for communicating with the modular communication hub 203, as shown in Figures 5 and 6.

[0103] Figure 8 illustrates a surgical instrument or tool equipped with multiple motors that can be activated to perform various functions. In a particular example, the first motor can be activated to perform a first function, the second motor can be activated to perform a second function, the third motor can be activated to perform a third function, the fourth motor can be activated to perform a fourth function, and so on. In a particular example, the multiple motors of the robotic surgical instrument 600 can be activated individually to produce firing, closing, and / or jointing motions in the end effector. The firing, closing, and / or jointing motions can be transmitted to the end effector, for example, via a shaft assembly.

[0104] In certain examples, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably coupled to a firing motor drive assembly 604, which can be configured to transmit the firing motion generated by the motor 602 to an end effector, specifically to displace an I-beam element. In certain examples, the firing motion generated by the motor 602 may, for example, deploy a staple from a staple cartridge into tissue captured by the end effector and / or advance the cutting edge of the I-beam element to cut the captured tissue. The I-beam element can be retracted by reversing the direction of the motor 602.

[0105] In certain examples, the surgical instrument or surgical tool may include a closing motor 603. The closing motor 603 may be operably coupled to a closing motor drive assembly 605, which may be configured to specifically displace a closing tube to close the anvil and transmit the closing motion generated by the motor 603 to the end effector to compress tissue between the anvil and the staple cartridge. The closing motion allows the end effector to transition from an open configuration to an approach configuration, for example, to capture tissue. The end effector may be moved to an open position by reversing the direction of the motor 603.

[0106] In certain examples, a surgical instrument or tool may include, for example, one or more articulation motors 606a, 606b. The motors 606a, 606b may be operably coupled to corresponding articulation motor drive assemblies 608a, 608b, which may be configured to transmit the articulation generated by the motors 606a, 606b to an end effector. In certain examples, the articulation may cause, for example, the end effector to articulate relative to the shaft.

[0107] As described herein, a surgical instrument or surgical tool may include multiple motors that can be configured to perform various independent functions. In certain examples, multiple motors of a surgical instrument or tool can be activated individually or separately to perform one or more functions while other motors remain stopped. For example, articulation motors 606a and 606b can be activated to articulate an end effector while firing motor 602 remains stopped. Alternatively, firing motor 602 can be activated to fire multiple staples and / or advance a cutting edge while articulation motor 606 remains stopped. Furthermore, a closure motor 603 may be activated simultaneously with firing motor 602 to advance the closure tube and I-beam element distally, as will be described in more detail below herein.

[0108] In certain examples, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In certain examples, the common control module 610 may correspond to one of the multiple motors at a time. For example, the common control module 610 may be individually connectable to and disconnectable from multiple motors of a robotic surgical instrument. In certain examples, multiple motors of a surgical instrument or tool may share one or more common control modules, such as the common control module 610. In certain examples, multiple motors of a surgical instrument or tool can engage with the common control module 610 individually and selectively. In certain examples, the common control module 610 can selectively switch between interfacing with one of the multiple motors of the surgical instrument or tool and interfacing with another of the multiple motors of the surgical instrument or tool.

[0109] In at least one embodiment, the common control module 610 can be selectively switched between an operable engagement with the articulation motors 606a, 606b and an operable engagement with either the firing motor 602 or the closing motor 603. In at least one embodiment, as shown in Figure 8, the switch 614 can be moved or transitioned between multiple positions and / or states. For example, in a first position 616, the switch 614 may electrically couple the common control module 610 with the firing motor 602; in a second position 617, the switch 614 may electrically couple the common control module 610 with the closing motor 603; in a third position 618a, for example, the switch 614 may electrically couple the common control module 610 with the first articulation motor 606a; and in a fourth position 618b, the switch 614 may electrically couple the common control module 610 with the second articulation motor 606b. In certain examples, a separate common control module 610 may also be electrically coupled to the launch motor 602, the closing motor 603, and the joint motion motors 606a, 606b. In certain examples, the switch 614 may be a mechanical switch, an electromechanical switch, a solid switch, or any preferred switching mechanism.

[0110] Each of the motors 602, 603, 606a, and 606b may be equipped with a torque sensor for measuring the output torque on the motor shaft. The force on the end effector may be sensed in any conventional manner, such as by force sensors on the outside of the jaws or by torque sensors on the motors that actuate the jaws.

[0111] In various examples, as shown in Figure 8, the common control module 610 may include a motor driver 626 which may comprise one or more H-bridge FETs. The motor driver 626 may modulate the power transmitted from the power supply 628 to the motor coupled to the common control module 610, for example, based on input from a microcontroller 620 ("controller"). In certain examples, the microcontroller 620 can be used to determine the current drawn by the motor while it is coupled to the common control module 610, for example, as described herein.

[0112] In certain examples, the microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-temporary computer-readable media or memory units 624 ("memory"). In certain examples, the memory 624 may store various program instructions, which, when executed, cause the processor 622 to perform some of the functions and / or calculations described herein. In certain examples, one or more of the memory units 624 may be coupled to the processor 622, for example.

[0113] In certain examples, power supply 628 may be used to power, for example, a microcontroller 620. In certain examples, power supply 628 may include a battery (or "battery pack" or "power pack"), such as a lithium-ion battery. In certain examples, the battery pack may be configured to be removably attached to a handle to power a surgical instrument 600. A number of battery cells connected in series may be used as power supply 628. In certain examples, power supply 628 may be, for example, replaceable and / or rechargeable.

[0114] In various examples, the processor 622 can control the motor driver 626 to control the position, direction of rotation, and / or speed of a motor coupled to a common control module 610. In certain examples, the processor 622 can signal the motor driver 626 to stop and / or disable a motor coupled to a common control module 610. The term “processor,” as used herein, should be understood to include any suitable microprocessor, microcontroller, or other basic computing device that integrates the functions of a computer’s central processing unit (CPU) on one or up to several integrated circuits. A processor may be a versatile programmable device that accepts digital data as input, processes that data according to instructions stored in memory, and provides the results as output. It may have internal memory and therefore may be an embodiment of sequential digital logic. A processor may operate with numbers and symbols represented in binary.

[0115] The processor 622 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In a particular example, the microcontroller 620 may be, for example, the LM 4F230H5QR available from Texas Instruments. In at least one embodiment, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes, among other features readily available in the product datasheet, 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and one or more 12-bit ADCs with 12 analog input channels. Other microcontrollers may be readily substituted for use with module 4410. Therefore, this disclosure should not be limited to this context.

[0116] Memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600, which can be coupled to the common control module 610. For example, memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the joint movement motors 606a and 606b. Such program instructions can cause the processor 622 to control the firing function, the closing function, and the joint movement function according to input from an algorithm or control program of the surgical instrument or tool.

[0117] One or more mechanisms and / or sensors, such as sensor 630, can be used to alert the processor 622 to program instructions that should be used in a particular setting. For example, sensor 630 can alert the processor 622 to use program instructions related to the firing, closing, and joint movement of the end effector. In a particular example, sensor 630 may include a position sensor that can be used to sense the position of switch 614, for example. Thus, if the processor 622 detects, for example, via sensor 630 that switch 614 is in a first position 616, it can use a program instruction associated with the firing of the end effector's I-beam; if the processor 622 detects, for example, via sensor 630 that switch 614 is in a second position 617, it can use a program instruction associated with the closing of the anvil; and if the processor 622 detects, for example, via sensor 630 that switch 614 is in a third position 618a or a fourth position 618b, it can use a program instruction associated with the joint movement of the end effector.

[0118] Figure 9 illustrates a diagram of a situation-aware surgical system 5100 according to at least one aspect of the present disclosure. In some examples, the data source 5126 may include, for example, a modular device 5102 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 5122 (e.g., an EMR database containing patient records), and a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor). The surgical hub 5104 may be configured to derive contextual information about a surgical procedure from the data, for example, based on a particular combination of received data or a particular order in which data is received from the data source 5126. Contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This ability of the surgical hub 5104 to derive or infer information about surgical procedures from received data, as in some embodiments of the surgical hub 5104, is sometimes referred to as “situational awareness.” In one example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104 that derives contextual information related to surgical procedures from received data.

[0119] The situational awareness system of the surgical hub 5104 can be configured to derive contextual information from data received from the data source 5126 in various different ways. For example, the situational awareness system may include a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 5122, patient monitoring device 5124, and / or modular device 5102) with corresponding contextual information about surgical procedures. In other words, the machine learning system can be trained to accurately derive contextual information about surgical procedures from provided inputs. In an embodiment, the situational awareness system may include a lookup table that stores pre-characterized contextual information about surgical procedures in association with one or more inputs (or ranges of inputs) corresponding to the contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situational awareness system to control the modular device 5102. In the embodiment, context information received by the situation awareness system of the surgical hub 5104 can be associated with a specific control adjustment of one or more modular devices 5102, or a set of control adjustments. In the embodiment, the situation awareness system may include a further machine learning system, a lookup table, or other such system which, when given context information as input, generates or retrieves one or more control adjustments of one or more modular devices 5102.

[0120] The surgical hub 5104 incorporating a situational awareness system can bring many advantages to the surgical system 5100. One advantage may include improved interpretation of sensed and collected data, which improves processing accuracy during the course of a surgical procedure and / or the use of the data. Returning to the previous embodiment, the situational awareness surgical hub 5104 can determine what type of tissue is being operated on, and therefore, if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 5104 can accurately accelerate or decelerate the motor of the surgical instrument according to the type of tissue.

[0121] The type of tissue being operated on can affect the adjustments made to the compression rate and load threshold of surgical staple and cutting instruments for specific interstitial space measurement. The situational awareness surgical hub 5104 can estimate whether the surgical procedure being performed is a thoracic or abdominal procedure, thereby allowing the surgical hub 5104 to determine whether the tissue clamped by the end effector of the surgical staple and cutting instrument is the lung (in the case of a thoracic procedure) or the stomach (in the case of an abdominal procedure). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical staple and cutting instrument according to the type of tissue.

[0122] The type of body cavity being operated on during an air insufflation procedure can affect the function of the fume ventilator. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing air insufflation) and determine the type of procedure. Since certain types of procedures are generally performed in specific body cavities, the surgical hub 5104 can appropriately control the motor speed of the fume ventilator to match the body cavity being operated on. In this way, the situational awareness surgical hub 5104 can provide a consistent amount of fume exhaust for both thoracic and abdominal procedures.

[0123] The type of procedure being performed can affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopic procedures may require a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situational awareness surgical hub 5104 can determine what type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue shape for the surgical procedure. Furthermore, the situational awareness surgical hub 5104 can be configured to adjust the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument not simply per procedure, but throughout the course of the surgical procedure. The situational awareness surgical hub 5104 can determine which steps of the surgical procedure are being performed or will be performed, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type according to the steps of the surgical procedure.

[0124] In the embodiment, the surgical hub 5104 may also derive data from an additional data source 5126 to improve conclusions drawn from one data source 5126. The contextually aware surgical hub 5104 may enhance data received from the modular device 5102 with contextual information constructed from other data sources 5126 regarding the surgical procedure. For example, the contextually aware surgical hub 5104 may be configured to determine whether hemostasis has been achieved (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, video or image data may not be conclusive. Therefore, in one example, the surgical hub 5104 may be further configured to make a determination regarding the integrity of the staple line or tissue weld by comparing physiological measurements (e.g., blood pressure sensed by a BP monitor communicably connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably coupled to the surgical hub 5104). In other words, the context-aware system of the surgical hub 5104 can take physiological measurement data into consideration to provide additional context when analyzing visualization data. This additional context can be useful when the visualization data itself may not be conclusive or may be incomplete.

[0125] For example, the situational awareness surgical hub 5104 can proactively activate the generator to which the RF electrosurgical instrument is connected if it determines that the instrument will be needed in a subsequent step of the procedure. By proactively activating the energy source, the instrument can be made available for use as soon as the preceding step of the procedure is completed.

[0126] The situational awareness surgical hub 5104 can determine, according to the characteristics of the surgical site that the surgeon is expected to need to see, whether the current or subsequent steps of the surgical procedure require different views or magnifications on the display. The surgical hub 5104 can then appropriately and proactively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108) so that the display automatically adjusts throughout the surgical procedure.

[0127] The situational awareness surgical hub 5104 can determine which steps of a surgical procedure are being performed or will be performed next, and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to request specific information.

[0128] Errors can be checked during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 5104 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 5104 has determined to be being performed. In some examples, the surgical hub 5104 can be configured to compare a list of items for a procedure, and / or a list of devices paired with the surgical hub 5104, to a proposed or expected manifest of items and / or devices for a given surgical procedure. If any discontinuities exist between the lists, the surgical hub 5104 can be configured to provide a warning indicating that a particular modular device 5102, patient monitoring device 5124, and / or other surgical items are missing. In some examples, the surgical hub 5104 may be configured to determine the relative distance or relative position of a modular device 5102 and a patient monitoring device 5124, for example, via proximity sensors. The surgical hub 5104 can compare the relative positions of the devices to a proposed or predicted layout for a particular surgical procedure. If any discontinuity exists between the layouts, the surgical hub 5104 may be configured to provide a warning indicating that the current layout of the surgical procedure deviates from the proposed layout.

[0129] The situational awareness surgical hub 5104 can determine whether a surgeon (or other healthcare professional) has made an error or deviated from the expected course of action during the surgical procedure. For example, the surgical hub 5104 may be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the surgical procedure with the expected steps or instruments for the type of surgical procedure that the surgical hub 5104 has determined to be performed. In some examples, the surgical hub 5104 may be configured to provide warnings indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0130] Surgical instruments (and other modular devices 5102) may be adjusted to suit the specific context of each surgical procedure (e.g., adjustment for different tissue types) and may verify their behavior during the surgical procedure. The following steps, data, and display adjustments may be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the procedure.

[0131] Figure 10 illustrates an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each stage of the surgical procedure. The following description of the timeline 5200 illustrated in Figure 9 should also refer to Figure 9. The timeline 5200 can illustrate a typical sequence of steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy procedure, beginning with setting up the operating room and ending with transferring the patient to the postoperative recovery room. The contextually aware surgical hub 5104 can receive data from data source 5126 throughout the surgical procedure, including data generated each time healthcare professionals utilize the modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular device 5102 and other data sources 5126, and can continuously derive estimations (i.e., contextual information) about the procedure in progress as new data is received, such as which step of the procedure is being performed at any given time. The contextual awareness system of the surgical hub 5104 may, for example, record data about the procedure to generate a report, verify the steps being taken by the medical professional, provide data or prompts that may be relevant to a particular procedural step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions described herein.

[0132] As a first step 5202 in this exemplary procedure, hospital staff can retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic procedure. Secondly 5204, staff can scan incoming medical supplies for the procedure. The surgical hub 5104 can cross-reference the scanned supplies with a list of supplies that may be used in various types of procedures and confirm that the combination of supplies matches a thoracic procedure. Furthermore, the surgical hub 5104 may also determine that the procedure is not a wedge resection (because the incoming supplies either do not include specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection). Thirdly 5206, healthcare workers can scan the patient band via a scanner 5128 that is communicably connected to the surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data. Fourth, 5208, a medical professional turns on the assistive devices. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they may include a fume extractor, an air insulator, and a medical imaging device. Once activated, the assistive devices, which are modular devices 5102, can automatically pair with a surgical hub 5104 located within a specific vicinity of the modular devices 5102 as part of their initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular embodiment, the surgical hub 5104 can determine that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team.When the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which step of the surgical procedure the surgical team is performing. Fifth of 5210, personnel attach the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, it can confirm that the patient is in the operating room, for example, as described in process 5207. Sixth of 5212, medical personnel can induce anesthesia in the patient. The surgical hub 5104 can estimate that the patient is under anesthesia based on data from the modular device 5102 and / or the patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 5212 is completed, the preoperative portion of the lung segmentectomy procedure is completed and the surgical portion begins.

[0133] 7. 5214. The lungs of the patient being operated on may collapse (while ventilation is switched to the contralateral lung). The surgical hub 5104 can estimate from the ventilator data, for example, that the patient's lungs have collapsed. The surgical hub 5104 can compare the detection of lung collapse with the expected steps of the procedure (which can be accessed or read in advance), and thus estimate that the surgical portion of the procedure has begun, and that thereby causing lung collapse may be the first surgical step in this particular procedure. 8. 5216. A medical imaging device 5108 (e.g., a scope) can be inserted, and video footage from the medical imaging device can be started. The surgical hub 5104 can receive medical imaging device data (i.e., video or image data) through its connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has begun. Furthermore, the surgical hub 5104 can determine that a particular procedure being performed is a segmentectomy rather than a lobectomy (note that wedge resections have not been taken into consideration by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Contextual information regarding the type of procedure being performed can be determined in various ways using data from the medical imaging device 124 (Figure 2), for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structure, by monitoring the number or type of medical imaging device being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used. For example, one technique for performing a VATS lobectomy may position the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while one technique for performing a VATS segmentectomy may position the camera anteriorly in the intercostal space relative to the segmental fissure. The contextual recognition system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure.An exemplary technique for performing VATS lobectomy may utilize a single medical imaging device. An exemplary technique for performing VATS segmentectomy may utilize multiple cameras. An exemplary technique for performing VATS segmentectomy may utilize an infrared light source (which can be communicably coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure, but this is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0134] 5218 of Section 9, the surgical team may begin the incision phase of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out phase of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the above-described procedure phase) corresponds to the incision phase. 5220 of Section 10, the surgical team may proceed to the ligation phase of the procedure. The surgical hub 5104 may receive data from surgical stapling and cutting instruments indicating that an instrument is being emitted, and can therefore infer that the surgeon is ligating arteries and veins. As with the previous phase, the surgical hub 5104 can derive this inference by cross-referencing the received data from surgical stapling and cutting instruments with the read-out phase in the process. 5222 of Section 11, the segmental resection phase of the procedure may be performed. The surgical hub 5104 can infer that a surgeon is transversely incising parenchymal tissue based on data from surgical stapling and cutting instruments (including data from their cartridges). The cartridge data may correspond, for example, to the size or type of staples being fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples being fired is used for parenchymal tissue (or other similar tissue types), thereby allowing the surgical hub 5104 to infer that the segmental resection portion of the procedure is being performed. 12th 5224, a nodule incision procedure is performed. The surgical hub 5104 can infer that the surgical team is incising a nodule and performing a leak test based on data received from a generator indicating that an RF or ultrasonic instrument is being fired. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision procedure, thereby allowing the surgical hub 5104 to make this inference.It should be noted that different instruments are better suited to specific tasks, so surgeons should periodically switch between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasound) instruments depending on the specific step in the procedure. Thus, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which step of the procedure the surgeon is performing. Once step 12, 5224, is completed, incision closure and the postoperative portion of the procedure can be initiated.

[0135] Step 13, 5226, allows the patient's anesthesia to be reversed. The surgical hub 5104 may, for example, estimate that the patient is waking from anesthesia based on ventilator data (i.e., the patient's respiratory rate begins to increase). Finally, step 14, 5228, may be the step in which a medical professional removes various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 may estimate that the patient is being transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. As can be seen from this exemplary procedure description, the surgical hub 5104 may determine or estimate when each step of a given surgical procedure is occurring based on data received from various data sources 5126 that are communicably coupled to the surgical hub 5104.

[0136] As shown in the first step 5202 of the timeline 5200 in Figure 10, in addition to estimating the type of surgical procedure to be performed using patient data from the EMR database, the patient data can also be utilized by the situational awareness surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0137] Figure 11 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system may be configured to monitor and analyze data relating to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analytics system. Although described as a surgical system, the cloud-based analytics system is not necessarily limited to that and may generally be a cloud-based medical system. As shown in Figure 11, the cloud-based analytics system may comprise a plurality of surgical instruments 7012 (which may be the same as or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same as or similar to hub 106), and a surgical data network 7001 (which may be the same as or similar to network 201) for connecting the surgical hubs 7006 to a cloud 7004 (which may be the same as or similar to cloud 204). Each of the plurality of surgical hubs 7006 may be communicably coupled to one or more surgical instruments 7012. Hub 7006 may also be communicably coupled to a cloud 7004 of a computer-implemented interactive surgical system via network 7001. Cloud 7004 may be a remote centralized source of hardware and software for storing, manipulating, and communicating data generated based on the operation of various surgical systems. As shown in Figure 11, access to Cloud 7004 can be achieved via network 7001, which may be the Internet or another suitable computer network. A surgical hub 7006 that can be coupled to Cloud 7004 can be considered the client side of a cloud computing system (i.e., a cloud-based analytics system). Surgical instruments 7012 may be paired with the surgical hub 7006 for the control and implementation of various surgical procedures or operations described herein.

[0138] In addition, the surgical instrument 7012 may be equipped with transceivers for data transmission to and from the corresponding surgical hub 7006 (which may also be equipped with transceivers). The combination of the surgical instrument 7012 and the corresponding hub 7006 can indicate a specific location, such as an operating room within a medical facility (e.g., a hospital) for providing medical surgery. For example, the memory of the surgical hub 7006 can store location data. As shown in Figure 11, the cloud 7004 comprises a central server 7013 (which may be the same as or similar to the remote server 7013), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7006. The central server 7013 of the cloud 7004 collectively manages the cloud computing system, which includes monitoring requests from client surgical hubs 7006 and managing the processing capacity of the cloud 7004 to perform those requests. Each central server 7013 may comprise one or more processors 7008 coupled to a suitable memory device 7010, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 7010 may contain machine-executable instructions that, when executed, cause the processor 7008 to run a data analysis module 7034 for cloud-based data analysis, operation, suggestions, and other operations described below. Furthermore, the processor 7008 may run the data analysis module 7034 independently or in conjunction with a hub application running independently by the hub 7006. The central server 7013 may also comprise a database 2212 of aggregated medical data, which may reside in memory 2210.

[0139] Based on its connection to various surgical hubs 7006 via network 7001, cloud 7004 can aggregate data from various surgical instruments 7012 and specific data generated by their corresponding hubs 7006. Such aggregated data may be stored in the aggregated medical database 7012 of cloud 7004. Specifically, cloud 7004 can advantageously perform data analysis and operations on the aggregated data to derive insights and / or perform functions that individual hubs 7006 cannot achieve on their own. For this purpose, as shown in Figure 11, cloud 7004 and surgical hubs 7006 are communicatively coupled to transmit and receive information. The I / O interface 7006 is connected to multiple surgical hubs 7006 via network 7001. In this way, the I / O interface 7006 can be configured to transfer information between the surgical hubs 7006 and the aggregated medical data database 7011. Thus, the I / O interface 7006 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hub 7006. These requests may be sent to the hub 7006 via the hub application. The I / O interface 7006 may include one or more high-speed data ports, including a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 7004 to the hub 7006. The hub application server 7002 of the cloud 7004 may be configured to host and supply shared functions to a software application (e.g., a hub application) running on the surgical hub 7006. For example, the hub application server 7002 may manage requests from the hub application through the hub 7006, control access to the aggregated medical data database 7011, and perform load balancing. The data analysis module 7034 is described in more detail with reference to Figure 12.

[0140] The specific cloud computing system configurations described in this disclosure may be designed to address a variety of problems arising in the context of medical surgeries and procedures performed using medical devices such as surgical instruments 7012, 112, etc. In particular, surgical instrument 7012 may be a digital surgical device configured to interact with the cloud 7004 in order to implement techniques to improve surgical outcomes. Various surgical instruments 7012 and / or surgical hubs 7006 may include a touch-controlled user interface so that a clinician can control the manner of interaction between the surgical instruments 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0141] Figure 12 is a block diagram illustrating a functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analytics system may include a number of data analytics modules 7034 that can be executed by a processor 7008 of the cloud 7004 to provide data analytics solutions to problems that arise particularly in the medical field. As shown in Figure 12, the functionality of the cloud-based data analytics module 7034 may be supported via a hub application 7014 hosted by a hub application server 7002 that can be accessed on the surgical hub 7006. The cloud processor 7008 and the hub application 7014 may work together to execute the data analytics module 7034. An application program interface (API) 7016 can define a set of protocols and routines corresponding to the hub application 7014. In addition, the API 7016 can manage the storage of data in and retrieval of data from an aggregated medical database 7012 for the operation of the application 7014. The cache 7018 can also store data (for example, temporarily) and can be coupled to the API 7016 for more efficient retrieval of data used by the application 7014. The data analysis module 7034 in Figure 12 may include modules for resource optimization 7020, data collection and aggregation 7022, authorization and security 7024, control program updates 7026, patient outcome analysis 7028, recommendations 7030, and data classification and prioritization 7032. Other suitable data analysis modules may also be implemented by the cloud 7004 in several embodiments. In one embodiment, the data analysis module may be used for specific recommendations based on the analysis of trends, outcomes, and other data.

[0142] For example, the data collection and aggregation module 7022 may be used to generate self-describing data (e.g., metadata), which may include identifying prominent features or configurations (e.g., trends), managing redundant datasets, and storing data in paired datasets that can be grouped by surgery but do not necessarily correspond to actual surgical dates and surgeons. In particular, paired datasets generated from the operation of surgical instruments 7012 may involve applying a binary classification, such as bleeding or non-bleeding events. More generally, the binary classification may be characterized as either a desirable event (e.g., a successful surgical procedure) or an undesirable event (e.g., a surgical instrument 7012 misfired or misused). The aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Thus, the data collection and aggregation module 7022 can generate aggregated metadata or other organized data based on the raw data received from the surgical hub 7006. For this purpose, the processor 7008 can be operationally coupled to the hub application 7014 and the aggregated medical data database 7011 in order to execute the data analysis module 7034. The data collection and aggregation module 7022 may store the aggregated and organized data in the aggregated medical data database 2212.

[0143] The resource optimization module 7020 can be configured to analyze this aggregated data to determine the optimal use of resources for a particular healthcare facility or group of healthcare facilities. For example, the resource optimization module 7020 can determine the optimal reorder point for surgical staple fasteners 7012 for a group of healthcare facilities based on the corresponding predicted demand for those staple fasteners 7012. The resource optimization module 7020 can also evaluate the resource use or other operational configurations of various healthcare facilities to determine whether resource use can be improved. Similarly, the suggestion module 7030 can be configured to analyze the aggregated data from the data collection and aggregation module 7022 and provide suggestions. For example, the suggestion module 7030 can suggest to a healthcare facility (e.g., a healthcare service provider such as a hospital) that a particular surgical instrument 7012 should be upgraded to an improved version based, for example, on the error rate being higher than predicted. In addition, the suggestion module 7030 and / or the resource optimization module 7020 can suggest better supply chain parameters, such as product reorder points, and provide suggestions for different surgical instruments 7012, their use, or procedures to improve surgical outcomes. Medical facilities can receive such suggestions via the corresponding surgical hub 7006. More specific suggestions regarding the parameters or configurations of various surgical instruments 7012 can also be provided. The hub 7006 and / or the surgical instrument 7012 may each have a display screen that shows data or suggestions provided by the cloud 7004.

[0144] The patient outcome analysis module 7028 can analyze surgical outcomes associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 may also analyze and evaluate other potential operating parameters. In this regard, the suggestion module 7030 may use these other potential operating parameters to make suggestions based on the resulting better surgical outcomes, such as better sealing or less bleeding. For example, the suggestion module 7030 may send suggestions to the surgical instrument 7006 regarding the timing of using a particular cartridge with the corresponding stapled surgical instrument 7012. Thus, the cloud-based analysis system may be configured to analyze large amounts of collected raw data while controlling for common variables and to provide centralized suggestions across multiple healthcare facilities (favorably determined based on aggregated data). For example, the cloud-based analysis system can analyze, evaluate, and / or aggregate things such as the type of medical procedure, the type of patient, the number of patients, and geographical similarities among healthcare providers using similar types of instruments in a way that a single healthcare facility cannot analyze independently. The control program update module 7026 can be configured to implement various suggestions for the surgical instrument 7012 when the corresponding control program is updated. For example, the patient outcome analysis module 7028 can identify correlations linking specific control parameters to successful (or unsuccessful) outcomes. Such correlations can be addressed when the updated control program is transmitted to the surgical instrument 7012 via the control program update module 7026. Updates to the instrument 7012, which can be transmitted via the corresponding hub 7006, may incorporate aggregated performance data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. In addition, the patient outcome analysis module 7028 and the suggestion module 7030 can identify improved ways of using the instrument 7012 based on the aggregated performance data.

[0145] The cloud-based analytics system may include security features implemented by Cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other preferred security credentials. These credentials may be stored in memory 7010 and associated with permitted cloud access levels. For example, based on providing accurate credentials, a surgical hub 7006 may be granted access to communicate with the cloud to a predetermined extent (e.g., to send or receive certain defined types of information). For this purpose, the aggregated medical data database 7011 of Cloud 7004 may include a database of certified credentials to verify the accuracy of the provided credentials. Different credentials may be associated with various levels of authorization for interaction with Cloud 7004, such as a predetermined access level for receiving data analysis generated by Cloud 7004. Furthermore, for security purposes, the cloud may maintain a database of hubs 7006, instruments 7012, and other devices, which may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be permitted to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be prevented from fully functioning when paired with the corresponding hub 7006. Additionally or alternatively, the cloud 7004 may flag instruments 7012 based on incompatibility or other specified criteria. In this way, counterfeit medical devices and the improper reuse of such devices across the cloud-based analysis system can be identified and addressed.

[0146] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal that may represent, for example, authorization credentials for access to the corresponding hub 7006 and the cloud 7004. A wired transceiver can also be used to transmit a signal. Such authorization credentials can be stored in the respective memory devices of the surgical instrument 7012. The authorization and security module 7024 can determine whether the authorization credentials are accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credentials can also be encrypted, for example, by using hash-based encryption. Upon transmitting appropriate authorization, the surgical instrument 7012 may signal to the corresponding hub 7006 and ultimately the cloud 7004 to indicate that the instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may transition to a state in which it is ready to receive medical data for storage in the aggregated medical data database 7011. This readiness for data transmission can be indicated, for example, by an optical indicator on the instrument 7012. Cloud 7004 can also send signals to surgical instruments 7012 to update their associated control programs. Cloud 7004 can send signals directed to specific categories of surgical instruments 7012 (e.g., electrosurgical instruments) to ensure that software updates for control programs are sent only to the appropriate surgical instruments 7012. Furthermore, Cloud 7004 may be used to implement system-wide solutions to address local or global issues based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, Cloud 7004 may modify the authorization credentials corresponding to this group to implement operational lockout for that group.

[0147] A cloud-based analytics system can enable monitoring of multiple healthcare facilities (e.g., healthcare facilities such as hospitals) to determine improved practices and propose changes accordingly (e.g., via the suggestion module 2030). Thus, the processor 7008 of Cloud 7004 can analyze data associated with individual healthcare facilities to identify facilities and aggregate that data with other data associated with other healthcare facilities. Groups may be defined, for example, based on similar operational practices or geographical location. In this way, Cloud 7004 can provide a wide range of analytics and suggestions to healthcare facility groups. The cloud-based analytics system can also be used for enhanced contextual awareness. For example, the processor 7008 may predictively model the effect of cost and effectiveness suggestions on a particular facility (compared to overall operations and / or various medical procedures). The cost and effectiveness associated with that particular facility can also be compared to the corresponding local areas of other facilities or any other equivalent facilities.

[0148] The data classification and prioritization module 7032 may prioritize and classify data based on severity (e.g., the severity, unexpectedness, or suspiciousness of the medical event associated with the data). This classification and prioritization may be used in conjunction with the functionality of other data analysis modules 7034 described herein to improve the cloud-based analysis and operation described herein. For example, the data classification and prioritization module 7032 may assign priorities to data analyses performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels may result in specific responses from the cloud 7004 (corresponding to the level of urgency), such as increased priority for rapid response, special processing, exclusion from the aggregated medical data database 7011, or other preferred responses. Furthermore, if necessary, the cloud 7004 may send requests (e.g., push messages) via the hub application server for additional data from the corresponding surgical instrument 7012. Push messages may result in notifications displayed on the corresponding hub 7006 to request support or additional data. This push message may be needed when the cloud detects a significant anomaly or outlier and the cloud is unable to determine the cause of that anomaly. The central server 7013 can be programmed to trigger this push message in certain critical situations, such as when data is determined to differ from a predicted value by a predetermined threshold, or when security is suspected to be involved.

[0149] Further illustrative details regarding the various functions described are provided in the following description. Each of the various descriptions may utilize a cloud architecture, as shown in Figures 11 and 12 as an example of hardware and software implementation.

[0150] Figure 13 illustrates a block diagram of a computer-implemented adaptive surgical system 9060 configured to adaptively generate control program updates for modular devices 9050, according to at least one aspect of the present disclosure. In some examples, the surgical system may include a surgical hub 9000, a plurality of modular devices 9050 communicably coupled to the surgical hub 9000, and an analysis system 9100 communicably coupled to the surgical hub 9000. While a single surgical hub 9000 may be illustrated, it should be noted that the surgical system 9060 may include any number of surgical hubs 9000 that can be connected to form a network of surgical hubs 9000 communicably coupled to the analysis system 9010. In some examples, the surgical hub 9000 may include a processor 9010 coupled to a memory 9020 and executing instructions stored therein, and a data relay interface 9030 to which data is transmitted to the analysis system 9100. In some examples, the surgical hub 9000 may further include a user interface 9090 having an input device 9092 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 9094 (e.g., a display screen) for providing output to the user. The output may include data from queries entered by the user, suggestions for products or combinations of products for use in a given procedure, and / or instructions for actions to be taken before, during, or after a surgical procedure. The surgical hub 9000 may further include an interface 9040 for communicatively connecting modular devices 9050 to the surgical hub 9000. In one embodiment, the interface 9040 may include a transceiver that can be communicatively connected to the modular devices 9050 via a wireless communication protocol. The modular devices 9050 may include, for example, surgical staple fasteners and cutters, electrosurgical instruments, ultrasound instruments, injectors, ventilators, and display screens. In some examples, the surgical hub 9000 can be further communicatively coupled to one or more patient monitoring devices 9052, such as an EKG monitor or a BP monitor.In some examples, the surgical hub 9000 can further be communicably connected to one or more databases 9054 or external computer systems, such as the EMR database of the medical facility where the surgical hub 9000 is located.

[0151] When the modular device 9050 is connected to the surgical hub 9000, the surgical hub 9000 can sense or receive perioperative data from the modular device 9050 and then associate the received perioperative data with surgical procedure outcome data. The perioperative data can indicate how the modular device 9050 was controlled during the course of the surgical procedure. Procedure outcome data includes data associated with the outcome from the surgical procedure (or its process), which may include whether the surgical procedure (or its process) had a positive or negative outcome. For example, outcome data may include whether the patient suffered postoperative complications from a particular procedure, or whether there was leakage (e.g., bleeding or air leakage) at a particular staple or incision line. The surgical hub 9000 can acquire surgical procedure outcome data by receiving data from an external source (e.g., from the EMR database 9054), by directly detecting outcomes (e.g., via one of the connected modular devices 9050), or by estimating the occurrence of outcomes through a situational awareness system. For example, data on postoperative complications can be read from the EMR database 9054, and data on staple or incision line leakage can be directly detected or estimated by the situational awareness system. Surgical procedure outcome data can be estimated by the situational awareness system from data received from various data sources, including the modular device 9050 itself, the patient monitoring device 9052, and the database 9054 to which the surgical hub 9000 is connected.

[0152] The surgical hub 9000 can transmit data and outcome data from associated modular devices 9050 to an analysis system 9100 for processing. By transmitting both perioperative data and procedural outcome data that show how the modular devices 9050 are controlled, the analysis system 9100 can correlate different modes of control of the modular devices 9050 with surgical outcomes for specific procedure types. In some examples, the analysis system 9100 may include a network of analysis servers 9070 configured to receive data from the surgical hub 9000. Each analysis server 9070 may include memory and a memory-coupled processor, which executes instructions stored in memory to analyze the received data. In some examples, the analysis servers 9070 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 9100 can then learn the optimal or preferred operating parameters for various types of modular devices 9050, generate adjustments to the control programs of the modular devices 9050 in the field, and then transmit (or "extrude") the updates to the control programs of the modular devices 9050.

[0153] Further details regarding the computer-implemented interactive surgical system 9060, including the surgical hub 9000 and various modular devices 9050 that can be connected thereto, are described in relation to Figures 5 and 6.

[0154] Figure 14 provides a surgical system 6500 according to the present disclosure, which may include a surgical instrument 6502 capable of communicating with a console 6522 or a portable device 6526 via a local area network 6518 or a cloud network 6520 via a wired or wireless connection. In various embodiments, the console 6522 and the portable device 6526 may be any suitable computing device. The surgical instrument 6502 may include a handle 6504, an adapter 6508, and a loading unit 6514. The adapter 6508 is releasably coupled to the handle 6504, and the loading unit 6514 is releasably coupled to the adapter 6508, so that the adapter 6508 transmits force from the drive shaft to the loading unit 6514. The adapter 6508 or the loading unit 6514 may include force gauges (not explicitly shown) disposed therein for measuring the force exerted on the loading unit 6514. The loading unit 6514 may include an end effector 6530 having a first jaw 6532 and a second jaw 6534. The loading unit 6514 may also be a multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without requiring the loading unit 6514 to be removed from the surgical site for reloading.

[0155] The first jaw 6532 and the second jaw 6534 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 6532 may be configured to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) which may be configured to fire at least one fastener multiple times, or which may be fired two or more times before being replaced. The second jaw 6534 may include an anvil which deforms the fasteners around the tissue or otherwise secures them around the tissue as the fasteners are ejected from the multi-fire fastener cartridge.

[0156] The handle 6504 may include a motor coupled to the drive shaft so as to affect the rotation of the drive shaft. The handle 6504 may include a control interface for selectively starting the motor. The control interface may include buttons, switches, levers, sliders, touch screens, and any other suitable input mechanisms or user interfaces, which can be engaged by a clinician to start the motor.

[0157] The control interface of the handle 6504 may communicate with the controller 6528 of the handle 6504 to selectively activate the motor and affect the rotation of the drive shaft. The controller 6528 may be located within the handle 6504 and is configured to receive input from the control interface and adapter data from the adapter 6508 or loading unit data from the loading unit 6514. The controller 6528 may analyze the input from the control interface and the data received from the adapter 6508 and / or loading unit 6514 to selectively activate the motor. The handle 6504 may also include a display that can be viewed by a clinician while the handle 6504 is in use. The display may be configured to show a portion of the adapter or loading unit data before, during, or after firing the instrument 6502.

[0158] The adapter 6508 may include an adapter identification device 6510 disposed therein, and the loading unit 6514 may include a loading unit identification device 6516 disposed therein. The adapter identification device 6510 may communicate with the controller 6528, and the loading unit identification device 6516 may communicate with the controller 6528. It will be understood that the loading unit identification device 6516 may communicate with the adapter identification device 6510 to relay or pass communication from the loading unit identification device 6516 to the controller 6528.

[0159] The adapter 6508 may also include a plurality of sensors 6512 (one shown) disposed around it to detect various conditions of the adapter 6508 or the environment (e.g., when the adapter 6508 is connected to a loading unit, when the adapter 6508 is connected to a handle, when the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside the adapter 6508, the number of times the adapter 6508 has fired, the peak force of the adapter 6508 during firing, the total amount of force applied to the adapter 6508, the peak recoil force of the adapter 6508, the number of pauses of the adapter 6508 during firing, etc.). The plurality of sensors 6512 can provide input to the adapter identification device 6510 in the form of data signals. The data signals from the plurality of sensors 6512 may be stored in the adapter identification device 6510 or used to update adapter data stored in the adapter identification device 6510. The data signals from the plurality of sensors 6512 may be analog or digital. Multiple sensors 6512 may include force gauges for measuring the force exerted on the loading unit 6514 during firing.

[0160] The handle 6504 and adapter 6508 can be configured to interconnect the adapter identification device 6510 and the loading unit identification device 6516 with the controller 6528 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). Additionally or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductively transmitting). It is also intended that the adapter identification device 6510 and the controller 6528 may be able to wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0161] The handle 6504 may include a transmitter 6506 configured to transmit instrument data from the controller 6528 to other components of the system 6500 (e.g., LAN 6518, cloud 6520, console 6522, or portable device 6526). The transmitter 6506 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 6500. For example, the controller 6528 may transmit instrument data to the console 6528 including the serial number of a connection adapter attached to the handle 6504 (e.g., adapter 6508), the serial number of a loading unit attached to the adapter (e.g., loading unit 6514), and the serial number of a multi-shot fastener cartridge loaded in the loading unit (e.g., multi-shot fastener cartridge). The console 6522 may then send back data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, to the controller 6528. The controller 6528 can display a message on the local device display, or send a message via the transmitter 6506 to the console 6522 or portable device 6526, which can then display the message on the display 6524 screen or the portable device screen, respectively.

[0162] Figure 15A illustrates an exemplary flow for determining the operating mode and operating in the determined mode. Computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems may be configured to be updatable. Such updates may include the inclusion of features and benefits not available to the user prior to the update. These updates can be established by any method of hardware, firmware, and software updates suitable for introducing features to the user. For example, computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems can be updated using interchangeable / swappable (e.g., hot-swappable) hardware components, flashable firmware devices, and updatable software systems.

[0163] Renewal can be conditional on any preferred criteria or set of criteria. For example, renewal can be conditional on one or more hardware capabilities of the system, such as processing power, bandwidth, or resolution. For example, renewal can be conditional on one or more software aspects, such as the purchase of a certain software code. For example, renewal can be conditional on a purchased service tier. A service tier may represent one and / or a set of features that a user has eligibility to use in connection with a computer-implemented interactive surgical system. A service tier can be determined by a license code, e-commerce server authentication interaction, hardware key, username / password combination, biometric authentication interaction, public / private key exchange interaction, etc.

[0164] In 10704, system / device parameters may be identified. System / device parameters may be any element or set of elements on which an update is conditional. For example, a computer-implemented interactive surgical system may detect a specific bandwidth of communication between a modular device and a surgical hub. For example, a computer-implemented interactive surgical system may detect instructions for purchasing a specific service tier.

[0165] In 10708, the operating mode may be determined based on identified system / device parameters. This determination may be performed by a process that maps system / device parameters to operating modes. The process may be a manual process and / or an automated process. The process may be the result of local computation and / or remote computation. For example, client / server interaction may be used to determine the operating mode based on identified system / device parameters. For example, local software and / or locally embedded firmware may be used to determine the operating mode based on identified system / device parameters. For example, a hardware key, such as a secure microprocessor, may be used to determine the operating mode based on identified system / device parameters.

[0166] In 10710, operation may proceed according to the determined operating mode. For example, the system or device may proceed to operate in the default operating mode. For example, the system or device may proceed to operate in an alternative operating mode. The operating mode may be indicated by control hardware, firmware, and / or software already residing in the system or device. The operating mode may also be indicated by newly installed / updated control hardware, firmware, and / or software.

[0167] Figure 15B illustrates an exemplary functional block diagram for changing the operating mode. The upgradeable element 10714 may include an initialization component 10716. The initialization component 10716 may include any hardware, firmware, and / or software suitable for determining the operating mode. For example, the initialization component 10716 may be part of a system or device startup procedure. The initialization component 10716 may be involved in interactions for determining the operating mode of the upgradeable element 10714. For example, the initialization component 10716 may interact with, for example, a user 10730, an external resource 10732, and / or a local resource 10718. For example, the initialization component 10716 may receive a license key from user 10730 to determine the operating mode. The initialization component 10716 may query an external resource 10732, such as a server, using the serial number of the upgradeable device 10714 to determine the operating mode. For example, the initialization component 10716 may query the local resource 10718, such as a local query to determine the amount of available bandwidth, and / or a local query for a hardware key to determine the operating mode.

[0168] An upgradeable element 10714 may include one or more operational components 10720, 10722, 10726, 10728 and an operational pointer 10724. The initialization component 10716 can instruct the operational pointer 10724 to direct the operation of the upgradeable element 10741 to the operational components 10720, 10722, 10726, 10728 corresponding to the determined operational mode. The initialization component 10716 may also instruct the operational pointer 10724 to direct the operation of the upgradeable element to the default operational component 10720. For example, the default operational component 10720 may be selected when no other alternative operational modes have been determined. For example, the default operational component 10720 may be selected in the event of an initialization component failure and / or interaction failure. The initialization component 10716 may instruct the operation pointer 10724 to direct the resident operation component 10722 to operate the upgradeable element 10714. For example, a particular feature may be present in the upgradeable element 10714, but requires startup to operate. The initialization component 10716 may instruct the operation pointer 10724 to direct the operation of the upgradeable element 10714 to install a new operation component 10728 and / or a newly installed operation component 10726. For example, new software and / or firmware may be downloaded. The new software and / or firmware may include code that enables the features represented by the selected operating mode. For example, a new hardware component may be installed to enable the selected operating mode.

[0169] As described herein, a hierarchical multi-display control scheme may provide various communication control options for medical professional-controlled secondary displays and primary operating room displays. For example, a powered surgical tool may operably communicate with a local display in the operating room outside the sterile field and at least one primary monitor to display multiple data and / or imaging sources. The local display may be interactable by the surgeon within the sterile field. Displays outside the sterile field, such as a primary external display, may show images in the form of a laparoscope and may contain other superimposed data streams from other devices other than the scope. A secondary display may be used to orient its displayed content onto the primary display or to remove it from the display. The added or removed data streams may originate from, pass through, or network with the secondary display. The secondary display may be part of a surgical instrument, such as a powered surgical tool. The secondary display may be a dedicated display system controlled by a medical professional within the sterile field.

[0170] Figure 19 shows an exemplary flow 18200 for operation under a hierarchical multi-display control mode. As shown in Figure 19, in 18205, a powered surgical device or a device such as a surgical hub may be connected to one or more displays inside the surgical sterile field and one or more displays outside the surgical sterile field. The powered surgical device may be, for example, the surgical instrument 112 shown in Figure 1, the surgical instrument 600 shown in Figure 8, or the modular device 5102 shown in Figure 9, or may include them. The powered surgical device may include a communication array operably connected to the displays inside and outside the surgical sterile field.

[0171] The display outside the sterile field may be, or include, a non-sterile display 107 or 109 as shown in Figure 2. For example, the display inside the surgical sterile field may be, or include, a secondary display such as a local display or a display on a surgical instrument. The display inside the surgical sterile field may be a secondary display. A healthcare professional can control the secondary display. The primary and secondary displays may have multiple levels of communication with the primary hub system. Embodiments of the primary and secondary displays can be found in more detail in U.S. Patent Application No. 15 / 940,671, “SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER,” filed March 29, 2018 (Agent Reference No. END8502USNP), which is incorporated herein by reference in its entirety.

[0172] Figure 16 illustrates an exemplary primary display 6200 associated with a surgical hub 206, comprising a global display window 6202 and a local instrument display window 6204, according to one aspect of the present disclosure. Referring subsequently to Figures 1 to 11, which show interaction with an interactive surgical system 100 environment including surgical hubs 106, 206, and Figures 12 to 14, which show instruments connected to the surgical hubs together, the behavior of the local instrument display 6204 may be displayed when an instrument 235 senses the connectable presence of the global display window 6202 via the surgical hub 206. The global display window 6202 may, for example, show a field of view 6206 of the surgical site 6208 at the center of a surgical hub display 215, also referred to herein as a monitor, as seen through a medical imaging device such as a laparoscope / endoscopy 219 coupled to an imaging module 238. The end effector portion 6218 of the connected instrument 235 may be shown within the field of view 6206 of the surgical site 6208 in the global display window 6202. The image shown on the display 237 located on the instrument 235 coupled to the surgical hub 206 is shown, for example, on the local instrument display window 6204 located in the lower right corner of the monitor 6200, or mirrored, as shown in Figure 16.

[0173] During operation, the relevant instruments, information, and menus may be displayed on the display 237 located on the instrument 235 until the instrument 235 senses its connection to the surgical hub 206, at which point all or some subsets of the information presented on the instrument display 6200 may be displayed (for example, only on) the local instrument display window 6204 portion of the surgical hub display 237 via the surgical hub 206. The information displayed on the local instrument display window 6204 may be mirrored on the display 237 located on the instrument 235, or may no longer be accessible on the explosive screen of the instrument display 237. This technique releases the instrument 235 to display different information on the surgical hub display 6200 or to display information in a larger font.

[0174] The primary display 6200 may provide intraoperative visualization of the surgical site 6208. High-level imaging may identify and visually highlight important structures such as the ureter 6220 (or nerves, etc.) 6222, tracking instrument proximity indicators 6210 which may be shown to the left of the display 6200. In the exemplary embodiment, instrument proximity indicators 6210 may indicate instrument-specific settings. For example, the top instrument proximity display 6212 may indicate settings for a unipolar instrument, the middle instrument proximity display 6214 may indicate settings for a bipolar instrument, and the bottom instrument proximity display 6212 may indicate settings for an ultrasound instrument.

[0175] The secondary displays may include independent secondary displays and / or dedicated local displays, which may be linked to the surgical hub 206 to provide an interaction portal via a touchscreen display and / or a secondary screen capable of displaying and providing status for any number of surgical hub 206 tracking data feeds. The secondary displays may display force to fire (FTF), interstitial space, power level, impedance, tissue compressive stability (creep), etc., while the primary displays may display only variables important to keep the feed clutter-free. Interactive displays may be used to move the display of specific information to the primary displays in a desired position, size, color, etc. In the exemplary embodiment, the secondary displays may display an instrument proximity indicator 6210 to the left of display 6200. A local instrument indicator 6204 is located on the lower right side of display 6200. The local instrument display 6204 presented on the surgical hub display 6200 may display icons for the end effector 6218, such as an icon for the staple cartridge 6224 currently in use, the size 6226 of the staple cartridge 6224 (e.g., 60 mm), and an icon for the current position of the end effector knife 6228.

[0176] The secondary display may be a display 237 as shown in Figure 5. Referring to Figure 5, the display 237 located on the instrument 235 can display the wireless or wired mounting of the instrument 235 to the surgical hub 206, as well as the instrument's communications and / or recordings on the surgical hub 206. Settings may be provided on the instrument 235 to allow the user to choose to mirror or extend the displays on both monitoring devices. The instrument control unit may be used to interact with the surgical hub display of the information supplied on the instrument. The instrument 235 may include a wireless communication circuit for wireless communication with the surgical hub 206, as described herein.

[0177] A first instrument coupled to the surgical hub 206 can be paired with the screen of a second instrument coupled to the surgical hub 206, allowing both instruments to display some hybrid combination of information from both devices, mirroring a portion of the primary display. The primary display 6200 of the surgical hub 206 can provide a 360° composite top view of the surgical site 6208 to avoid collateral structures. For example, a secondary display of an end-effector surgical stapler may be provided within the primary display 6200 of the surgical hub 206, or on a separate display, to provide better perspective around areas within the current field of view 6206.

[0178] Figure 17 illustrates an exemplary primary display having a composite overhead view of the end effector 6234 portion of a surgical stapler mapped using two or more imaging arrays or one array and time, to provide multiple perspective views of the end effector 6234 to enable composite imaging of the overhead field of view. The techniques described herein can be applied to ultrasound instruments, electrosurgical instruments, combinations of ultrasound / electrosurgical instruments, and / or combinations of surgical staplers / electrosurgical instruments. Several techniques can be employed to overlay or extend images and / or text from multiple image / text sources to present a composite image on a display (e.g., a single display).

[0179] As shown in Figure 17, the primary display 6200 of the surgical hub 206 can display a primary window 6230. The primary window 6230 may be located in the center of the screen and shows an enlarged or exploded narrow-angle view of the surgical field 6232. The primary window 6230 located in the center of the screen shows an enlarged or narrow-angle view of the end effector 6234 of a surgical stapler that grasps a blood vessel 6236. The primary window 6230 can display a composite image to generate a composite image that enables visualization of structures adjacent to the surgical field 6232. A second window 6240 may be shown in the lower left corner of the primary display 6200. The second window 6240 displays a composite image in a wide-angle view at standard focus of the image shown in the primary window 6230 in an overhead view. The overhead view provided in the second window 6240 allows the observer to easily view items outside the narrow-field surgical field 6232 without moving the laparoscope or other imaging devices 239 coupled to the imaging module 238 of the surgical hub 206. The third window 6242 may be shown in the lower right corner of the primary display 6200 and shows an icon 6244 representing the staple cartridge of the end effector 6234 (for example, a staple cartridge in this embodiment), along with additional information such as "4 rows" indicating the number of staple rows 6246 and "35 mm" indicating the distance the knife has traveled along the length of the staple cartridge 3248. Below the third window 6242 is an icon 6258 of the frame showing the current status of the clamp stabilization sequence, indicating clamp stabilization.

[0180] The local display / secondary display may be or include an augmented reality (AR) device. The AR device may include a head-mounted display (HMD). The HMD may include a processor, a non-temporary computer-readable memory storage medium, and executable instructions contained within the storage medium that are executable by the processor to perform the method or part of the method disclosed herein. The HMD may include a graphics processor for rendering 2D or 3D video and / or images for display.

[0181] Figure 18 illustrates a perspective view showing a surgeon using a surgical instrument, including a handle assembly housing and a wireless circuit board, during a surgical procedure while wearing a pair of safety glasses. The safety glasses may be, or may include, an augmented reality (AR) device capable of functioning as a secondary display. The wireless circuit board transmits signals to the safety glasses worn by the surgeon using the surgical instrument during the procedure. The signals are received by a wireless port on the safety glasses. One or more illumination devices on the front lens of the safety glasses change color, extinguish, or emit light in response to the received signals to indicate information about the status of the surgical instrument to the surgeon. The illumination devices may be positioned on the periphery of the front lens so as not to obstruct the surgeon's line of sight. Further embodiments are described in U.S. Patent No. 9,011,427, “SURGICAL INSTRUMENT WITH SAFETY GLASSES,” issued April 21, 2015, which is incorporated herein by reference in its entirety.

[0182] Figure 18 shows one version of safety glasses 6991 that may be worn by a surgeon 6992 during a surgical procedure while using a medical device. When in use, a wireless communication board housed within a surgical instrument 6993 may communicate with a wireless port 6994 on the safety glasses 6991. The exemplary surgical instrument 6993 is a battery-powered device, although the instrument 6993 may be powered by a cable or other means. The instrument 6993 includes an end effector. In particular, the wireless communication board 6995 transmits one or more wireless signals, indicated by arrows (B, C), to the wireless port 6994 of the safety glasses 6991. The safety glasses 6991 receive the signals, analyze the received signals, and display the indicated status information received by the signals on the lenses 6996 for the user, such as the surgeon 6992 wearing the safety glasses 6991. Additionally or alternatively, the wireless communication board 6995 transmits a wireless signal to the surgical monitor 6997, as described above, so that the surgical monitor 6997 can display the received indicated status information to the surgeon 6992.

[0183] One version of the safety glasses 6991 may include a lighting device on the periphery of the safety glasses 6991. The lighting device provides peripheral visual sensory feedback of the device 6993, thereby allowing the safety glasses 6991 to communicate with the user wearing the safety glasses 6991. The lighting device may be, for example, a light-emitted diode ("LED"), a series of LEDs, or any other suitable lighting device known to those skilled in the art and evident from the teachings herein.

[0184] Referring back to Figure 19, at 18210, one or more multi-display control parameters may be obtained. At 18215, the current multi-display control mode may be identified based on the multi-display control parameters.

[0185] For example, the current multi-display control mode can be selected from a number of multi-display control modes that are pre-configured, dynamically updated, semi-dynamically updated, periodically updated, or can be pre-set. The multi-display control mode can support or disable various multi-display control capabilities as described herein.

[0186] Multi-display control parameters may include, but are not limited to, system capabilities such as hardware capabilities, firmware capabilities, and / or software capabilities associated with the surgical device and / or system. For example, if a secondary display lacks the hardware capability to receive user instructions, the surgical hub may switch to a multi-display control mode in which it can disable control of the display content of the primary display via the secondary display.

[0187] Multi-display control parameters may include consumer control parameters such as subscription levels. For example, a healthcare facility may purchase a subscription to multi-display control functionality. Some subscription levels may provide displays with access to surgical data collected from external systems (e.g., via hubs and / or surgical instruments), while other subscription levels may restrict display control and connectivity to internal devices.

[0188] Multi-display control parameters may include available data bandwidth, power capacity and usage, processor and memory utilization, and / or internal or auxiliary systems.

[0189] In the embodiment, the multi-display control parameters may be instructions from or include instructions from the surgical hub. The multi-display control parameters may include instructions from the hierarchical system. Referring to Figures 22 to 24, the hierarchical system may scale display connectivity and control communication between the surgical hub 18706, local display 18725, and main display 18730, and communication between the surgical hub 18706 and external servers 18713 / 18722, etc., based on available data bandwidth, power capacity and usage, processor and memory utilization, and / or internal or auxiliary systems. The hierarchical system may determine the maximum communication capacity under which the surgical system can operate.

[0190] For example, if the hierarchical system detects that the power capacity associated with an operating room, surgical hub, and / or medical facility is below a threshold, it may scale down the multi-display control capacity. For example, if the hierarchical system detects other system conditions that may ensure a scale down of multi-display control capacity, such as the available data bandwidth falling below a threshold, memory utilization exceeding a certain threshold, power consumption exceeding a certain threshold, and / or other system conditions, it may restrict or disable interactions between primary and secondary displays, interactions between the surgical hub and displays, and / or communication between surgical systems (such as surgical instruments in the surgical hub and / or OR) and external servers. For example, a sterile field display-based control mode 18800 (as shown in Figure 23) may be scaled down to a one-way communication mode 18700 (as shown in Figure 22). External communication capabilities (as described herein with reference to Figure 24) may be disabled. In embodiments, the hierarchical system may be a module within the surgical hub or a system outside the surgical hub.

[0191] In 18220, it may be determined whether to generate visualization control data associated with the display outside the surgical sterile field, based on the current multi-display control mode. In 18230, the device may interact with the display inside the surgical sterile field and the display outside the surgical sterile field, based on the determination.

[0192] The surgical system may operate under various multi-display control modes, such as one-way communication mode, sterile field display-based control mode, and / or remote aggregation analysis mode.

[0193] Figure 22 shows an exemplary multi-display control mode, such as one-way communication mode 18700. As shown, the surgical instrument 18712 and the local display 18725 may be located in the sterile field. The local display 18725 may be part of the surgical instrument 18712 or outside of the surgical instrument 18712. The main monitor 18730 may be a primary display as described herein and may be located outside the sterile field. The surgical hub 18706 may be located outside the sterile field, as shown. In some embodiments, the surgical hub 18706 may be located inside the sterile field. The local display 18725 can receive information from the surgical instrument 18712 for display and / or receive information from the surgical hub 18706 for display. The information from the surgical instrument 18712 and the information from the surgical hub 18706 may be combined for display on the local display 18725. The local display 18725 can provide a location where the surgeon can view or access a portion of the overall data environment.

[0194] As shown in Figure 22, the surgical instrument 18712 may receive content from the surgical hub 18706 for display on the local display 18725 inside the surgical sterile field, and transmit the received content to the display 18725. In some embodiments, the local display 18725 can receive display content from the surgical hub 18706. The surgical hub 18706 can transmit the display content to the main monitor 18730. In the exemplary one-way communication mode 18700, the generation of visualization control data associated with the display outside the surgical sterile field based on instructions received via the local display 18725 may be disabled. The generation of aggregate analysis requests may be disabled.

[0195] Figure 23 shows an exemplary multi-display control mode 18800 that may support sterile field display-based control. As shown, surgical instruments 18712 and local displays 18725 may be located in the sterile field. Local displays 18725 may be part of surgical instruments 18712 or outside of surgical instruments 18712. The main monitor 18730 may be a primary display as described herein and may be located outside the sterile field. In some embodiments, the main display may be an outside sterile field display such as displays 107 or 109 as shown in Figure 2, or may include one. In some embodiments, the main display may be an inside sterile field display such as a primary display 119 as shown in Figure 2, or may include one. The surgical hub 18706 may be located outside the sterile field as shown. In some embodiments, the surgical hub 18706 may be located inside the sterile field. As shown in Figure 23, the surgical instrument 18712 may receive content from the surgical hub 18706 for display on the local display 18725 inside the surgical sterile field, and transmit the received content to the display 18725. In some embodiments, the local display 18725 may receive display content from the surgical hub 18706. The surgical hub 18706 may transmit the display content to the main monitor 18730.

[0196] As shown in Figure 23, the surgical instrument 18712 can receive visualization control data from the local display 18725 inside the surgical sterile field to control the main display 18730. The surgical instrument 18712 can transmit visualization control data to the main display 18730 to control the main display 18730. The surgical instrument 18712 can transmit visualization control data directly to the main display 18730. The surgical instrument 18712 can transmit visualization control data via the surgical hub 18706. The main display 18730 can adjust its display according to the visualization control data.

[0197] In some embodiments, the surgical hub 18706 can receive visualization control data from a local display 18725 inside the surgical sterile field to control the main display 18730. The surgical hub 18706 can transmit visualization control data to the main display 18730 to control the main display 18730. The main display 18730 can adjust its display based on the visualization control data.

[0198] For example, a secondary display, such as the local display 18725 shown in Figure 23, may serve as a user interface for displaying and controlling surgical hub functions from within the sterile field. The secondary display can be used to change the display location, what information is displayed where, and / or to divert control of a particular function or device.

[0199] During surgical procedures, surgeons may not have access to user interface devices and displays within the sterile field for interactive input by the surgeon. Therefore, surgeons may not need to interface with user interface devices and surgical hubs from within the sterile field, nor can they control other surgical devices through the surgical hub from within the sterile field.

[0200] The local display 18725 may include a display unit used within the sterile field, accessible for input and display by the surgeon, which may allow the surgeon to have interactive input control from the sterile field to control other surgical devices and / or displays coupled to a surgical hub. The display unit may be sterilized and located within the sterile field so as to allow the surgeon to interact with the display unit and the surgical hub to directly interact with instruments and configure instruments as needed without leaving the sterile field. The display unit may also be a master device and may be used for display, control, and tool control exchange, allowing the surgeon to receive feeds from other surgical hubs without leaving the sterile field.

[0201] The display unit may be an interactive touchscreen display, an interface configured to connect the interactive touchscreen display to a surgical hub, a processor, and memory coupled to the processor, or may include these. The memory can store instructions executable by the processor to receive input commands from the interactive touchscreen display located inside the sterile field, and can send input commands to the surgical hub to control devices coupled to the surgical hub located outside the sterile field.

[0202] The local display 18725 inside the surgical sterile field may also be a secondary surgeon display within the sterile field and may be accessible for input and display by the surgeon within the sterile field interactive control display. The sterile field interactive control display may be shared or dedicated to specific medical professionals.

[0203] The local display 18725 may be mounted on an operating table, a stand, or on the patient's abdomen or chest. The sterile field display 18725 is sterile and allows the surgeon to interact with the non-sterile field display 18730 and the surgical hub 18706 via the sterile field display 18725. This may provide the surgeon with control of the system and allow the surgeon to interface directly with and configure the non-sterile field display 18730 as needed (e.g., without leaving the sterile field). The sterile field display 18725 may be configured as a master device and may be used for display, control, and tool control exchange, and may allow feeding from other surgical hubs (e.g., without the surgeon leaving the sterile field).

[0204] Surgical instruments such as surgical instrument 18712 may include a local display 18725. Surgical instrument 18712 may include an interactive touchscreen display, an interface configured to connect the interactive touchscreen display to a surgical hub, and a control circuit configured to receive input commands from the interactive touchscreen display located inside the sterile field and to transmit the input commands to the surgical hub to control a device connected to the surgical hub located outside the sterile field.

[0205] The non-temporary computer-readable medium, when executed, can store computer-readable instructions that cause the machine to receive input commands from an interactive touchscreen display located inside the sterile field, and to send input commands to the surgical hub via an interface configured to connect the interactive touchscreen display to the surgical hub, thereby controlling a display connected to the surgical hub located outside the sterile field.

[0206] Designed for use within a sterile field and providing a display unit accessible for input and viewing by the surgeon, it offers the surgeon interactive input control from the sterile field to control other surgical displays coupled to a surgical hub.

[0207] A secondary user interface via a display unit may allow control of a non-sterile display from within the sterile field. The display unit may include a display device such as an iPad, e.g., a portable interactive touchscreen display device configured to be introduced into the operating room in a sterile manner. It may be pairable like any other device or may be position-sensitive. The display device may be enabled to function in this manner whenever the display device is positioned over a specific location on the patient's draped abdomen during a surgical procedure.

[0208] A local display 18725 inside the surgical sterile field may generate visualization control data to control non-sterile displays, such as a main display 18730 outside the surgical sterile field. For example, the visualization control data may indicate changes in display location and / or what information can be displayed where. The visualization control data may indicate deflecting control of a particular function or device.

[0209] For example, the local display 18725 may reconfigure the wireless activation device and its paired energy device in the operating room if the surgeon hands the device to another. The local display 18725 may be used as an interactable, scalable secondary display that allows the surgeon to overlay other feeds or images, such as a laser Doppler scanning array. The local display 18725 may be used to recall preoperative scans or images for review. Once the vascular pathway and depth, as well as the device trajectory, are estimated, the surgeon may use a sterile field interactable, scalable secondary display to overlay other feeds or images. An example of a sterile field display-based control is described in U.S. Patent Application No. 15 / 940,671, filed March 29, 2018, entitled "Surgical Hub with Direct Interface Control with Secondary Surgeon Display Units Designed within the Sterile Field and Accessible for Input and Display by the Surgeon," which is incorporated herein by reference in its entirety.

[0210] Figure 24 shows an exemplary multi-display control mode 18900 that can support remote data aggregation and analysis. As shown, a surgical instrument 18712 may receive visualization control data from a local display 18725 inside the surgical sterile field to control the main display 18730. The surgical instrument 18712 may transmit visualization control data to the main display 18730 (e.g., directly or via a surgical hub 18706) to control the main display 18730. The main display 18730 can adjust its display based on the visualization control data. In some embodiments, a surgical hub 18706 may receive visualization control data from a local display 18725 inside the surgical sterile field to control the main display 18730. The surgical hub 18706 can transmit visualization control data to the main display 18730 to control the main display 18730. The main display 18730 can adjust its display based on the visualization control data.

[0211] As shown in Figure 24, the local display 18725 may communicate with the remote server 18713 and / or the aggregate database 18722 through the surgical hub 18706. The local display 18725 may access and display data and / or analyses residing on the remote server 18713. The local display 18725 can combine the data and / or analyses read from the remote server 18713 with content and data created locally for display.

[0212] Figure 20 shows an exemplary flow 18300 for operation under a hierarchical multi-display control mode. As shown in Figure 20, in 18312, the current multi-display control mode may be identified based on the multi-display control parameters. In 18315, it may be determined whether the current multi-display control mode supports sterile field display-based control. If it is determined that the current multi-display control mode supports sterile field display-based control, in 18316, visualization control data associated with the display outside the surgical sterile field may be acquired. The visualization control data may also be acquired via the display inside the surgical sterile field, as described herein. If it is determined that the current multi-display control mode does not support sterile field display-based control, in 18318, the generation of visualization control data associated with the display outside the surgical sterile field may be disabled.

[0213] Figure 21 shows an exemplary flow 18400 for operation under a hierarchical multi-display control mode. As shown in Figure 21, in 18412, the current multi-display control mode may be identified based on one or more multi-display control parameters. In 18415, it may be determined based on the current multi-display control mode whether it supports aggregate analysis requests. Based on the determination that the current multi-display control mode supports aggregate analysis requests, in 18416, an aggregate analysis request may be generated.

[0214] For example, a request for aggregated analysis may include requests for historical datasets, usage processes, problem solving, images associated with the current procedure, videos associated with the current procedure, comparative information from previous patients existing in a similar format, and / or requests for procedure suggestions.

[0215] In 18418, an aggregated analysis response is obtained and, in 18420, may be combined with surgical data generated based on the sensed surgical data for display on a display inside the surgical sterile field. In 18428, the generation of an aggregated analysis request may be disabled based on the determination that the current multi-display control mode does not support aggregated analysis requests.

[0216] Examples of aggregation (e.g., remote aggregation), requirements, and analysis are described in detail in U.S. Patent Application No. 15 / 940,668, filed on March 29, 2018, with attorney reference number END8501USNP2, entitled “AGGREGATION AND REPORTING OF SURGICAL HUB DATA,” which is incorporated herein by reference in its entirety.

[0217] Figure 25 shows an exemplary flow 18500 for operation under a hierarchical multi-display control mode. In embodiments, the steps shown in Figure 25 may be performed by a surgical hub as described herein. As shown in Figure 25, in 18512, the current multi-display control mode may be identified based on one or more multi-display control parameters. In 18515, it may be determined whether the current multi-display control mode supports sterile field display-based control. If it is determined that the current multi-display control mode supports sterile field display-based control, then in 18516, visualization control data associated with one or more displays outside the surgical sterile field may be received from one or more displays inside the surgical sterile field. For example, visualization control data generated via a secondary display can be controlled to modify, focus, or control the data displayed on a display outside the sterile field. This allows healthcare professionals to view the data more seamlessly compared to other imaging or pre-operative imaging systems.

[0218] For example, sterile field display-based control may allow data to move back and forth relative to a surgical hub. A display within the sterile field may function as a control system that can control where information is displayed (e.g., on a display outside the sterile field), where information is stored, and so on.

[0219] In 18520, the display outside the surgical sterile field may be controlled based on visualization control data from the display inside the surgical sterile field. If it is determined that the current multi-display control mode does not support sterile field display-based control, in 18528, control of the display outside the surgical sterile field from the display inside the surgical sterile field may be disabled. In some embodiments, steps 18512, 18515, 18516, 18520, and 18528 may be performed by a surgical hub. In some embodiments, steps 18512, 18515, 18516, 18520, and 18528 may be performed by surgical instruments.

[0220] Figure 26 shows an exemplary flow 18600 for operation under a hierarchical multi-display control mode. In embodiments, the steps shown in Figure 25 may be performed by a surgical hub as described herein. As shown in Figure 26, in 18612, the current multi-display control mode may be identified based on the multi-display control parameters. In 18615, it may be determined based on the current multi-display control mode whether it supports remote aggregate analysis requests. Based on the determination that the current multi-display control mode supports remote aggregate analysis requests, in 18616, an aggregate analysis request may be generated and sent to a remote server.

[0221] For example, aggregate analysis requests may include requests for historical datasets, usage procedures, problem solving, images associated with the current procedure, videos associated with the current procedure, comparative information from previous patients existing in a similar format, and / or procedure suggestions. Aggregate analysis requests are generated based on instructions from a display inside the sterile field.

[0222] In 18618, an aggregated analysis response may be received from a remote server. The aggregated analysis response may correspond to an aggregated analysis request. For example, the aggregated analysis response may include, but is not limited to, a historical dataset, usage procedures, problem solving, images associated with the current procedure, videos associated with the current procedure, comparative information from previous patients existing in a similar format, and / or procedure suggestions. As illustrated, the received aggregated analysis response may be combined in 18620 with surgical data generated based on the sensed surgical data for display on a display inside the surgical sterile field. In 18628, the generation of an aggregated analysis request may be disabled based on the determination that the current multi-display control mode does not support remote aggregated analysis requests.

[0223] In some embodiments, steps 18612, 18615, 18616, 18618, 18620, and 18628 may be carried out by a surgical hub. In some embodiments, steps 18612, 18615, 18616, 18618, 18620, and 18628 may be carried out by surgical instruments. [Examples]

[0224] 1. A surgical instrument supplied with power, A communication array operably connected to, for example, a first display which may be located inside the surgical sterile field, and a second display which may be located outside the surgical sterile field, A processor, and the processor, To obtain multi-display control parameters, Identifying the current multi-display control mode based on multi-display control parameters, Based on the current multi-display control mode, determine whether to generate visualization control data associated with the second display, A powered surgical instrument configured to interact with a first display and a second display based on that determination.

[0225] 2. A powered surgical instrument as described in Example 1, wherein a communication array is operably connected to a surgical hub, and multi-display control parameters include instructions from the surgical hub.

[0226] 3. The communication array is operationally connected to the surgical hub, and the processor is: Receiving content from the surgical hub to be displayed on the first display, A powered surgical instrument according to Embodiment 1 or 2, further configured to transmit received content to a first display.

[0227] 4. The processor, Based on the determination that the current multi-display control mode supports sterile field display-based control, visualization control data associated with the second display is acquired via the first display, A powered surgical instrument according to any one of Examples 1 to 3, further configured to transmit visualization control data associated with a second display to a surgical hub in order to control a second display.

[0228] 5. The processor, A powered surgical instrument according to any one of Examples 1 to 4, further configured to disable the generation of visualization control data associated with a second display based on the determination that the current multi-display control mode does not support sterile field display-based control.

[0229] 6. The processor, Based on the determination that the current multi-display control mode supports sterile field display-based control, the system receives a user instruction to change the content on the second display, Based on the received user instructions, generate visualization control data associated with the second display, A powered surgical instrument according to any one of Examples 1 to 5, further configured to transmit visualization control data associated with a second display to a surgical hub in order to control a second display.

[0230] 7. The powered surgical instrument according to Embodiment 6, wherein a user instruction to change the content on the second display is received via the first display.

[0231] 8. User instructions, Projecting content associated with the first display onto the second display, or A powered surgical instrument according to Example 6 or 7, which includes at least one of the following: removing content associated with a first display from a second display.

[0232] 9. A powered surgical instrument is equipped with at least one sensor for sensing surgical data, and a processor is provided. Based on the current multi-display control mode, determine whether to request aggregated analysis from a remote server via the surgical hub, Based on the determination that aggregate analysis is required, an aggregate analysis request is generated, Receiving aggregated analysis responses via a communication array, A powered surgical instrument from any one of Examples 1 to 8, further configured to combine the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on a first display.

[0233] 10. A powered surgical instrument according to any one of Examples 1 to 9, wherein the powered surgical instrument comprises a first display inside the surgical sterile field.

[0234] 11. A powered surgical instrument as described in any one of Examples 1 to 10, wherein the first display inside the surgical sterile field is located outside the powered surgical instrument.

[0235] In all of the embodiments described above, where a first display is mentioned, it is preferably a first display inside the surgical sterile field. In all of the embodiments described above, where a second display is mentioned, it is preferably a second display outside the surgical sterile field.

[0236] 12. A surgical hub, A communication array operably connected to a first display, which may be located inside the surgical sterile field, and a second display, which may be located outside the surgical sterile field, A processor, and the processor, To obtain multi-display control parameters, Identifying the current multi-display control mode based on multi-display control parameters, Based on the current multi-display control mode, determine whether to receive visualization control data associated with the second display from the first display, A surgical hub is configured to interact with a first display and a second display based on that determination.

[0237] 13. The processor, Based on the determination that the current multi-display control mode supports sterile field display-based control, the system receives visualization control data associated with the second display, The surgical hub according to Embodiment 12 is further configured to control a second display based on received visualization control data associated with the second display.

[0238] 14. The processor, Based on the current multi-display control mode, determine whether to retrieve the aggregated analysis from the remote server for display on the first display, Based on the determination that the current multi-display control mode supports remote aggregation, an aggregation analysis request is generated, Receiving aggregated analysis responses via a communication array, The surgical hub according to Example 12 or 13, further configured to combine the received aggregated analysis response with surgical data received from at least one instrument in order to generate content to be displayed on a first display.

[0239] 15. A method for communicating with a first display inside a surgical sterile field and a second display outside a surgical sterile field, To obtain multi-display control parameters, Identifying the current multi-display control mode based on multi-display control parameters, Based on the current multi-display control mode, determine whether to generate visualization control data associated with a second display outside the surgical sterile field, A method comprising, based on that determination, interacting with a first display inside the surgical sterile field and a second display outside the surgical sterile field.

[0240] 16. Based on the determination that the current multi-display control mode supports sterile field display-based control, visualization control data associated with the second display outside the surgical sterile field is acquired via the first display inside the surgical sterile field, The method according to Example 15, further comprising transmitting visualization control data associated with a second display in order to control a second display outside the surgical sterile field.

[0241] 17. The method according to Example 15, further comprising disabling the generation of visualization control data associated with a second display outside the surgical sterile field, based on the determination that the current multi-display control mode does not support sterile field display-based control.

[0242] 18. Based on the determination that the current multi-display control mode supports sterile field display-based control, the system receives a user instruction to change the content on the second display outside the surgical sterile field, Based on the received user instructions, the system generates visualization control data associated with a second display outside the surgical sterile field, The method according to Example 15, further comprising transmitting visualization control data associated with a second display outside the surgical sterile field in order to control a second display outside the surgical sterile field.

[0243] 19. The method according to Example 18, wherein a user instruction to change the content on a second display outside the surgical sterile field is received via a first display inside the surgical sterile field.

[0244] 20. Based on the current multi-display control mode, determine whether to request aggregated analysis from the remote server, and Based on the determination that aggregate analysis is required, an aggregate analysis request is generated, Receiving aggregated analysis responses, Receiving detected surgical data from at least one sensor, The method according to Example 15, further comprising combining the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on a first display inside the surgical sterile field.

[0245] In the embodiments described above, including at least Embodiment 1, a powered surgical tool may be configured to selectively interact with a first display and a second display. For example, if the first display is inside the surgical sterile field and the second display is outside the surgical sterile field, this allows the powered surgical tool to selectively interact with displays in different parts of the operating room, which may be in the same room or different rooms. To ensure strict adherence to medical procedures concerning the sterile field, medical professionals may benefit from utilizing multiple displays located inside and outside the sterile field. The tool is configured such that selective interaction is controlled by multi-display control parameters. In embodiments, the multi-display control parameters may include one or more instructions, such as instructions from a surgical hub, consumer control parameters such as subscription levels, or instructions from a hierarchical system. In other embodiments, the multi-display control parameters may include software or hardware parameters such as available data bandwidth, power capacity and usage, and processor and memory utilization. These software or hardware parameters may belong to a powered surgical tool, a surgical hub, a first and / or second display, or any system between them.

[0246] Powered surgical devices can operate under various multi-display control modes, such as one-way communication mode, sterile field display-based control mode, and / or remote aggregation analysis mode. In at least one embodiment (3), the tool is configured to operate under one-way communication mode. When operating in an exemplary one-way communication mode, the surgical device can receive and transmit content to a display inside the surgical sterile field. Content to be displayed on the display inside the surgical sterile field may be received from a surgical hub. This can be useful for medical professionals, such as surgeons or nurses, within the sterile field to view information about the procedure being performed. In at least one embodiment (4) and one embodiment (6), the tool is configured to operate under sterile field display-based control mode. Embodiment 9 corresponds to remote aggregation analysis mode.

[0247] Each mode or hierarchy enables different levels of communication control tailored to the requirements associated with the mode of each hierarchy or display. Adaptive control allows surgeons to be provided with appropriate information on the display. For example, in the basic mode or hierarchy, the secondary display simply serves as a place where surgeons can view or access a portion of the overall data environment. More sophisticated modes or hierarchies allow the secondary display to function not only as another display but also as a control system, thereby enabling control over systems within the connected environment, the location of information display, and the location of information storage. The secondary display may further be a display portal for surgeons to access aggregated or compiled data on a remote server, where historical datasets, usage processes, problem solving, images or videos of value for the current procedure, comparative information from previous patients existing in a similar format, or procedure suggestions may reside. In this most complex mode or hierarchy of operation, the secondary display would communicate through data and analysis residing on a remote server within the local display, in combination with locally created content and data, enabling their access and display.

[0248] The various multi-display control modes described above provide hierarchical multi-display control with various communication control options for secondary and primary operating room displays controlled by medical professionals. Designed for use within a sterile field and accessible for input and display by the surgeon, the display unit provides the surgeon with interactive input control from the sterile field to control other surgical displays coupled to a surgical hub. A secondary user interface via the display unit may enable control of non-sterile displays from within the sterile field and may provide suitable visualization control. In the above embodiments, the level or hierarchy may be controlled or limited by hardware, the communication capabilities of the systems involved, or user input. Powered surgical instruments can adaptively control interaction with secondary displays inside the surgical sterile field and primary displays outside the surgical sterile field for hierarchical control. Hierarchical multi-display control allows powered surgical instruments and a surgical hub to adaptively control interaction with secondary displays in the sterile field. Through interaction with secondary and primary displays, the surgeon may be provided with a wide range of information and enhanced control. This can further improve the quality and safety of the operation.

[0249] The methods of Examples 15-20 and the surgical hubs of Examples 12-14 correspond to the devices of Examples 1-11. Therefore, the above considerations regarding Examples 1-1 also apply to Examples 12-20.

[0250] The following list of embodiments forms part of this description.

[0251] 1. A powered surgical instrument comprising a first display inside a surgical sterile field, a communication array operably connected to a second display outside the surgical sterile field, and a processor, the processor being configured to: obtain multi-display control parameters; identify a current multi-display control mode based on the multi-display control parameters; determine whether to generate visualization control data associated with the second display outside the surgical sterile field based on the current multi-display control mode; and interact with the first display inside the surgical sterile field and the second display outside the surgical sterile field based on the determination.

[0252] 2. The powered surgical instrument according to embodiment 1, wherein the communication array is operably connected to a surgical hub and the multi-display control parameters include an instruction from the surgical hub.

[0253] 3. The powered surgical instrument according to embodiment 1, wherein the communication array is operably connected to a surgical hub, and the processor is further configured to receive content for display on the first display inside the surgical sterile field from the surgical hub and transmit the received content to the first display.

[0254] 4. The powered surgical instrument according to embodiment 1, wherein the processor is further configured to obtain visualization control data associated with the second display outside the surgical sterile field via the first display inside the surgical sterile field based on a determination that the current multi-display control mode supports sterile field display-based control, and transmit the visualization control data associated with the second display to the surgical hub to control the second display outside the surgical sterile field.

[0255] 5. The processor is further configured to invalidate the generation of visualization control data associated with a second display outside the surgical sterile field based on a determination that the current multi-display control mode does not support control of the sterile field display base, the powered surgical instrument according to Embodiment 1.

[0256] 6. The processor is further configured to receive a user instruction to change the content on a second display outside the surgical sterile field based on a determination that the current multi-display control mode supports control of the sterile field display base, generate visualization control data associated with the second display outside the surgical sterile field based on the received user instruction, and transmit the visualization control data associated with the second display outside the surgical sterile field to a surgical hub to control the second display outside the surgical sterile field, the powered surgical instrument according to Embodiment 1.

[0257] 7. The powered surgical instrument according to Embodiment 6, wherein a user instruction to change the content on a second display outside the surgical sterile field is received via a first display inside the surgical sterile field.

[0258] 8. The powered surgical instrument according to Embodiment 6, wherein the user instruction indicates at least one of projecting the content associated with the first display inside the surgical sterile field onto the second display outside the surgical sterile field or removing the content associated with the first display inside the surgical sterile field from the second display outside the surgical sterile field.

[0259] 9. The powered surgical instrument according to Embodiment 1, further configured to perform the following: a powered surgical instrument comprising at least one sensor for sensing surgical data, and a processor determining whether to request aggregate analysis from a remote server via a surgical hub based on the current multi-display control mode; generating an aggregate analysis request based on the determination to request aggregate analysis; receiving an aggregate analysis response via a communication array; and combining the received aggregate analysis response with surgical data generated based on the sensed surgical data for display on a first display inside the surgical sterile field.

[0260] 10. The powered surgical instrument according to Embodiment 1, wherein the powered surgical instrument has a first display inside the surgical sterile field.

[0261] 11. The powered surgical instrument according to Embodiment 1, wherein the first display inside the surgical sterile field is located outside the powered surgical instrument.

[0262] 12. A surgical hub comprising: a communication array operably connected to surgical instruments, a first display inside a surgical sterile field, and a second display outside a surgical sterile field; and a processor, wherein the processor is configured to acquire multi-display control parameters; identify the current multi-display control mode based on the multi-display control parameters; determine, based on the current multi-display control mode, whether to receive visualization control data associated with the second display outside the surgical sterile field from the first display inside the surgical sterile field; and interact with the first display inside the surgical sterile field and the second display outside the surgical sterile field based on that determination.

[0263] 13. The surgical hub according to Embodiment 1, further configured to: receive visualization control data associated with a second display outside the surgical sterile field based on a determination that the current multi-display control mode supports sterile field display-based control; and control the second display outside the surgical sterile field based on the received visualization control data associated with the second display outside the surgical sterile field.

[0264] 14. The surgical hub according to Embodiment 1, further configured to: determine whether to read aggregate analysis from a remote server for display on a first display inside the surgical sterile field, based on the current multi-display control mode; generate an aggregate analysis request, based on the determination that the current multi-display control mode supports remote aggregation; receive an aggregate analysis response via a communication array; and combine the received aggregate analysis response with surgical data received from at least one instrument to generate content for display on the first display inside the surgical sterile field.

[0265] 15. A method for communicating with a first display inside a surgical sterile field and a second display outside a surgical sterile field, comprising: obtaining multi-display control parameters; identifying a current multi-display control mode based on the multi-display control parameters; determining whether to generate visualization control data associated with a second display outside a surgical sterile field based on the current multi-display control mode; and interacting with the first display inside a surgical sterile field and the second display outside a surgical sterile field based on the determination.

[0266] 16. The method according to Embodiment 15, further comprising: obtaining visualization control data associated with a second display outside the surgical sterile field via a first display inside the surgical sterile field, based on the determination that the current multi-display control mode supports sterile field display-based control; and transmitting visualization control data associated with the second display to control the second display outside the surgical sterile field.

[0267] 17. The method according to Embodiment 15, further comprising disabling the generation of visualization control data associated with a second display outside the surgical sterile field, based on the determination that the current multi-display control mode does not support sterile field display-based control.

[0268] 18. The method of Embodiment 15, further comprising: receiving a user instruction to change the content on a second display outside the surgical sterile field based on a determination that the current multi-display control mode supports sterile field display-based control; generating visualization control data associated with the second display outside the surgical sterile field based on the received user instruction; and transmitting the visualization control data associated with the second display outside the surgical sterile field to control the second display outside the surgical sterile field.

[0269] 19. The method according to Embodiment 18, wherein a user instruction to change the content on a second display outside the surgical sterile field is received via a first display inside the surgical sterile field.

[0270] 20. The method of Embodiment 15, further comprising: determining whether to request aggregate analysis from a remote server based on the current multi-display control mode; generating an aggregate analysis request based on the determination to request aggregate analysis; receiving an aggregate analysis response; receiving sensed surgical data from at least one sensor; and combining the received aggregate analysis response with surgical data generated based on the sensed surgical data for display on a first display inside the surgical sterile field.

[0271] [Implementation Method] (1) A surgical instrument that is powered by electricity, A communication array operably connected to, for example, a first display which may be located inside the surgical sterile field, and a second display which may be located outside the surgical sterile field, A processor and a processor, the processor To obtain multi-display control parameters, Based on the aforementioned multi-display control parameters, the current multi-display control mode is identified, Based on the current multi-display control mode, it is determined whether to generate visualization control data associated with the second display, A powered surgical instrument configured to interact with the first display and the second display based on the aforementioned determination. (2) The powered surgical instrument according to Embodiment 1, wherein the communication array is operably connected to a surgical hub, and the multi-display control parameters include instructions from the surgical hub. (3) The communication array is operably connected to the surgical hub, and the processor is Receiving content from the surgical hub for display on the first display, The powered surgical instrument according to embodiment 1 or 2, further configured to perform: transmitting the received content to the first display. (4) The processor is Based on the determination that the current multi-display control mode supports the control of the sterile field display base, acquiring the visualization control data associated with the second display via the first display, The powered surgical instrument according to any one of embodiments 1 to 3, further configured to perform: transmitting the visualization control data associated with the second display to the surgical hub to control the second display. (5) The processor is Based on the determination that the current multi-display control mode does not support the control of the sterile field display base, further configured to invalidate the generation of the visualization control data associated with the second display. The powered surgical instrument according to any one of embodiments 1 to 4.

[0272] (6) The processor is Based on the determination that the current multi-display control mode supports the control of the sterile field display base, receiving a user instruction to change the content on the second display, Generating the visualization control data associated with the second display based on the received user instruction, The powered surgical instrument according to any one of embodiments 1 to 5, further configured to perform: transmitting the visualization control data associated with the second display to the surgical hub to control the second display. (7) The powered surgical instrument according to embodiment 6, wherein the user instruction to change the content on the second display is received via the first display. (8) The user instruction is Projecting the content associated with the first display onto the second display, or A powered surgical instrument according to embodiment 6 or 7, which instructs at least one of the following: removing the content associated with the first display from the second display. (9) The powered surgical instrument is equipped with at least one sensor for sensing surgical data, and the processor is Based on the current multi-display control mode, it is determined whether to request aggregated analysis from a remote server via the surgical hub, Based on the determination that aggregate analysis is required, an aggregate analysis request is generated, Receiving aggregated analysis responses via the aforementioned communication array, A powered surgical instrument according to any one of embodiments 1 to 8, further configured to combine the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on the first display. (10) The powered surgical instrument according to any one of embodiments 1 to 9, wherein the powered surgical instrument comprises the first display inside the surgical sterile field.

[0273] (11) The powered surgical instrument according to any one of embodiments 1 to 10, wherein the first display inside the surgical sterile field is located outside the powered surgical instrument. (12) A surgical hub, A communication array operably connected to a surgical instrument, for example, a first display which may be located inside the surgical sterile field, and a second display which may be located outside the surgical sterile field, A processor and a processor, the processor To obtain multi-display control parameters, Based on the aforementioned multi-display control parameters, the current multi-display control mode is identified, Based on the current multi-display control mode, it is determined whether to receive visualization control data associated with the second display from the first display, A surgical hub configured to interact with the first display and the second display based on the aforementioned determination. (13) The processor Based on the determination that the current multi-display control mode supports sterile field display-based control, the visualization control data associated with the second display is received, The surgical hub according to Embodiment 12, further configured to control the second display based on the received visualization control data associated with the second display. (14) The processor Based on the current multi-display control mode, it is determined whether to read the aggregated analysis from the remote server for display on the first display, Based on the determination that the current multi-display control mode supports remote aggregation, an aggregation analysis request is generated, Receiving aggregated analysis responses via the aforementioned communication array, A surgical hub according to embodiment 12 or 13, further configured to combine the received aggregated analysis response with surgical data received from at least one instrument in order to generate content to be displayed on a first display. (15) A method for communicating with a first display inside a surgical sterile field and a second display outside the surgical sterile field, To obtain multi-display control parameters, Based on the aforementioned multi-display control parameters, the current multi-display control mode is identified, Based on the current multi-display control mode, determine whether to generate visualization control data associated with the second display outside the surgical sterile field, A method comprising interacting with the first display inside the surgical sterile field and the second display outside the surgical sterile field based on the determination.

[0274] (16) Based on the determination that the current multi-display control mode supports sterile field display-based control, the visualization control data associated with the second display outside the surgical sterile field is acquired via the first display inside the surgical sterile field, The method of Embodiment 15, further comprising transmitting the visualization control data associated with the second display in order to control the second display outside the surgical sterile field. (17) The method of Embodiment 15, further comprising disabling the generation of the visualization control data associated with the second display outside the surgical sterile field, based on the determination that the current multi-display control mode does not support sterile field display-based control. (18) Receiving a user instruction to change the content on the second display outside the surgical sterile field, based on the determination that the current multi-display control mode supports sterile field display-based control, Based on the received user instructions, the visualization control data associated with the second display outside the surgical sterile field is generated, The method of Embodiment 15, further comprising transmitting the visualization control data associated with the second display outside the surgical sterile field in order to control the second display outside the surgical sterile field. (19) The method according to embodiment 18, wherein the user instruction to change the content on the second display outside the surgical sterile field is received via the first display inside the surgical sterile field. (20) Based on the current multi-display control mode, determine whether to request aggregated analysis from a remote server, Based on the determination that aggregate analysis is required, an aggregate analysis request is generated, Receiving aggregated analysis responses, Receiving detected surgical data from at least one sensor, The method of Embodiment 15, further comprising combining the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on the first display inside the surgical sterile field.

Claims

1. A powered surgical instrument, A communication array operably connected to a first display located inside the surgical sterile field and a second display located outside the surgical sterile field, A processor and a processor, the processor To obtain multi-display control parameters, Based on the aforementioned multi-display control parameters, the current multi-display control mode is identified, Based on the current multi-display control mode, it is determined whether to generate visualization control data associated with the second display, A powered surgical instrument configured to interact with the first display and the second display based on the determination.

2. The powered surgical instrument according to claim 1, wherein the communication array is operably connected to a surgical hub, and the multi-display control parameters include instructions from the surgical hub.

3. The communication array is operably connected to the surgical hub, and the processor is Receiving content from the surgical hub for display on the first display, A powered surgical instrument according to claim 1 or 2, further configured to transmit the received content to the first display.

4. The aforementioned processor, Based on the determination that the current multi-display control mode supports sterile field display-based control, the visualization control data associated with the second display is acquired via the first display, A powered surgical instrument according to claim 2 or 3, further configured to transmit the visualization control data associated with the second display to the surgical hub in order to control the second display.

5. The aforementioned processor, A powered surgical instrument according to any one of claims 1 to 4, further configured to disable the generation of the visualization control data associated with the second display based on the determination that the current multi-display control mode does not support sterile field display-based control.

6. The aforementioned processor, Based on the determination that the current multi-display control mode supports sterile field display-based control, the system receives a user instruction to change the content on the second display, Based on the received user instructions, the visualization control data associated with the second display is generated, A powered surgical instrument according to claim 2 or 3, further configured to transmit the visualization control data associated with the second display to the surgical hub in order to control the second display.

7. The powered surgical instrument according to claim 6, wherein the user instruction to change the content on the second display is received via the first display.

8. The aforementioned user instruction, Projecting the content associated with the first display onto the second display, or A powered surgical instrument according to claim 6 or 7, which instructs at least one of the following: removing the content associated with the first display from the second display.

9. The powered surgical instrument is equipped with at least one sensor for sensing surgical data, and the processor is Based on the current multi-display control mode, it is determined whether to request aggregated analysis from a remote server via the surgical hub, Based on the determination that aggregate analysis is required, an aggregate analysis request is generated, Receiving aggregated analysis responses via the aforementioned communication array, A powered surgical instrument according to claim 2 or 3, further configured to combine the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on the first display.

10. The powered surgical instrument according to any one of claims 1 to 9, wherein the powered surgical instrument comprises the first display located inside the surgical sterile field.

11. The powered surgical instrument according to any one of claims 1 to 10, wherein the first display inside the surgical sterile field is located outside the powered surgical instrument.

12. It is a surgical hub, A communication array operably connected to surgical instruments, a first display located inside the surgical sterile field, and a second display located outside the surgical sterile field, A processor and a processor, the processor To obtain multi-display control parameters, Based on the aforementioned multi-display control parameters, the current multi-display control mode is identified, Based on the current multi-display control mode, it is determined whether to receive visualization control data associated with the second display from the first display, A surgical hub configured to interact with the first display and the second display based on the aforementioned determination.

13. The aforementioned processor, Based on the determination that the current multi-display control mode supports sterile field display-based control, the visualization control data associated with the second display is received, The surgical hub according to claim 12, further configured to control the second display based on the received visualization control data associated with the second display.

14. The aforementioned processor, Based on the current multi-display control mode, it is determined whether to read the aggregated analysis from the remote server for display on the first display, Based on the determination that the current multi-display control mode supports remote aggregation, an aggregation analysis request is generated, Receiving aggregated analysis responses via the aforementioned communication array, The surgical hub according to claim 12 or 13, further configured to combine the received aggregate analysis response with surgical data received from the surgical instrument in order to generate content to be displayed on the first display.

15. A method for communicating with a first display inside a surgical sterile field and a second display outside the surgical sterile field, To obtain multi-display control parameters, Based on the aforementioned multi-display control parameters, the current multi-display control mode is identified, Based on the current multi-display control mode, determine whether to generate visualization control data associated with the second display outside the surgical sterile field, A method comprising interacting with the first display inside the surgical sterile field and the second display outside the surgical sterile field based on the determination.

16. Based on the determination that the current multi-display control mode supports sterile field display-based control, the visualization control data associated with the second display outside the surgical sterile field is acquired via the first display inside the surgical sterile field, The method of claim 15, further comprising transmitting the visualization control data associated with the second display in order to control the second display outside the surgical sterile field.

17. The method according to claim 15, further comprising disabling the generation of the visualization control data associated with the second display outside the surgical sterile field, based on the determination that the current multi-display control mode does not support sterile field display-based control.

18. Based on the determination that the current multi-display control mode supports sterile field display-based control, the system receives a user instruction to change the content on the second display outside the surgical sterile field, Based on the received user instructions, the visualization control data associated with the second display outside the surgical sterile field is generated, The method of claim 15, further comprising transmitting the visualization control data associated with the second display outside the surgical sterile field in order to control the second display outside the surgical sterile field.

19. The method according to claim 18, wherein the user instruction to change the content on the second display outside the surgical sterile field is received via the first display inside the surgical sterile field.

20. Based on the current multi-display control mode, determine whether to request aggregated analysis from a remote server, Based on the determination that aggregate analysis is required, an aggregate analysis request is generated, Receiving aggregated analysis responses, The system receives detected surgical data from at least one sensor, The method according to claim 15, further comprising combining the received aggregated analysis response with surgical data generated based on the sensed surgical data for display on the first display inside the surgical sterile field.

Images