Interactive information overlay on multiple surgical displays

Abstract

The surgical hub may communicate with at least one augmented reality device, at least one display, and at least one smart surgical device. The surgical hub may provide an interactive overlay of a display, such as a surgical display or a display on a smart surgical device. For example, the surgical hub may generate augmented reality (AR) content for overlaying on the display. The AR device may overlay or overlay an additional data set or data stream received from the hub on the display. This interactive overlay may allow a user of the AR device to layer data on a screen when the user is looking at the screen. The surgical hub may adjust the AR content based on the display the user is looking at through the AR device. For example, the hub may adjust the AR content when the user looks from one display to another.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application is related to the following applications filed simultaneously, and the content of each of these is incorporated herein by reference. · Attorney Docket No. END9287USNP1, Invention Title "METHOD FOR OPERATING TIERED OPERATION MODES IN A SURGICAL SYSTEM", · Attorney Docket No. END9287USNP14, Invention Title "TIERED SYSTEM DISPLAY CONTROL BASED ON CAPACITY AND USER OPERATION", · Attorney Docket No. END9287USNP15, Invention Title "COOPERATIVE SURGICAL DISPLAYS", and · Attorney Docket No. END9287US17, Invention Title "COMMUNICATION CONTROL OPTIONS FOR A SURGEON CONTROLLED SECONDARY DISPLAY AND PRIMARY DISPLAY".

Background Art

[0002] Surgical systems often incorporate imaging systems that allow clinicians to view the surgical site and / or one or more parts thereof on one or more displays, such as monitors. These displays may be located locally at the surgical site and / or remotely. The imaging system may include a scope equipped with a camera that views the surgical site and transmits the view to a display visible to the clinician. Examples of scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and exoscopy. The imaging system may be limited by the information that can be recognized by and / or communicated to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery by certain imaging systems. Additionally, certain imaging systems may not be able to communicate and / or transmit certain information to the clinician during surgery. [Overview of the project] [Means for solving the problem]

[0003] According to embodiments of the present invention, a surgical hub may be operably connected to at least one display, an augmented reality (AR) device, and at least one surgical instrument. The surgical hub may obtain a data stream from the surgical instrument for display on the display. The surgical hub may generate AR content for overlaying on the display. For example, the AR content may be generated based on the user role and / or the display (e.g., the display type and / or the content displayed on the display). The surgical hub may be configured to determine whether the user of the AR device is looking at the display, and if it detects that the user of the AR device is looking at the display, it may overlay the AR content on the display via the AR device. If it detects that the user of the AR device is no longer looking at the display, the surgical hub may stop supplying the AR content associated with the display via the AR device. The surgical hub may generate other AR content for overlaying on a second display. The surgical hub may determine whether the user of the AR device is looking at the second display, and if it detects that the user of the AR device is looking at the second display, it may overlay the generated AR content for overlaying on the second display via the AR device.

[0004] AR devices can adjust the display data on a monitor or device screen. For example, a surgical device can receive display control indicators from an AR device and adjust the content to be displayed on the display based on the received display control indicators.

[0005] According to one embodiment of the present invention, a surgical hub may be configured to receive images from a surgical scope and surgical information from at least one surgical instrument. The surgical hub may be operably connected to multiple displays, such as multiple AR devices, a primary display and a secondary display. The surgical hub may acquire AR control parameters and, based on the AR control parameters, may determine AR content to overlay on a data stream displayed on the display. The surgical hub may transmit the AR content to a first AR device. The surgical hub may determine different AR content to overlay on the data stream. The surgical hub may transmit this AR content to a second AR device. The AR content may include steps for using a surgical instrument, device settings, device status, device commands for use, operating parameters, indicators of detected anomalies, preoperative imaging, intraoperative imaging, instrument data, and / or treatment commands. The AR content may show anatomical information such as tumor markings, tumor margins, and blood flow information (e.g., high blood flow, low blood flow).

[0006] As illustrated above and below, the surgical hub enables the same screen to display different content to different members of the surgical team through AR devices. Specifically, the AR content is overlaid on the display for the user of the AR device and is not visible to members of the surgical team who are not using that AR device. Through the use of multiple AR devices displaying different AR content, different AR content can be overlaid on the same screen for different users. By using one or more AR devices in this way, it becomes possible to directly transmit information to members of the surgical team who need it, without distracting other members of the team who do not need the information. This faster access to information and less distraction leads to higher efficiency, improved patient safety, and improved surgical outcomes. Overlaying this information directly on the first display using AR devices leads to further improvements, as the first data stream and the AR content are displayed in the same location. This allows both data sources to be viewed together, and therefore more quickly, and allows the AR content to interact with the first data stream, such as labeling items shown in the first data stream or highlighting aspects of its content. This allows coordinated information about the progress of surgery to be sent to the surgical team. AR content can assist the surgical team in performing specific steps of the surgical procedure, ensuring correct device setup and use, guaranteeing that the procedure is working as intended, and detecting and correcting abnormalities or anomalies in the surgical procedure. By displaying predicted outcomes of specific surgical actions as part of the AR content, the surgical team can be supported in the surgery being performed, the surgical action can be executed to higher standards of accuracy and safety, potentially fatal errors can be avoided or minimized, and surgical outcomes can be significantly improved. The use of AR control parameters such as surgical progress, user orientation, and surgical context helps the AR content to be updated in real time and to explain events specific to the current surgical procedure, such as any anomalies.Therefore, overall, the displayed AR content further improves efficiency, patient safety, and surgical outcomes.

[0007] For example, AR control parameters may include, or may not include, the user's orientation to the display. The surgical hub may detect whether the user of the AR device is looking at the display. In response to detecting that the user is looking at the display, the surgical hub may generate and / or send AR content to the AR device for overlaying on the display. When the surgical hub detects that the user of the AR device is no longer looking at the display, it may stop supplying the AR content associated with the display via the AR device. By supplying AR content via the AR device, the user can see a composite image in which the content on the display is combined with the AR content supplied via the AR device. The surgical hub may instruct the AR device to detect whether the user of the AR device is looking at the display, and in response to the detection, to overlay one or more data layers of overlays onto the content displayed on the display via the AR device.

[0008] For example, AR control parameters may include, or may not include, user roles associated with the AR device. The surgical hub may be configured to generate multiple data layers for inclusion in AR content. In some examples, the data layers may be associated with one or more user roles. The surgical hub can identify a first user role associated with a first AR device and, based on the first user role, determine a first overlay dataset for inclusion in the first AR content. The surgical hub may identify a second user role associated with a second AR device and, based on the second user role, determine a second different overlay dataset for inclusion in the second AR content. The overlay dataset may include one or more data layers generated by the surgical hub.

[0009] For example, AR control parameters may include a user role associated with the AR device and a display type associated with the display. For example, a surgical hub may identify a user role associated with the AR device and a display type associated with the display. Based on the display type and user role, the surgical hub may determine AR content to overlay on the content displayed on the display via the AR device. The display type may be an instrument display located on a smart surgical instrument, a shared display in the operating room, or a personal display.

[0010] A hub communicating with an AR device can provide a simulation or confirmation of an intended action. AR content may include indicators of predicted outcomes if the user performs the intended action. For example, if a user clamps or holds jaws over an intended area to staple, cut, or seal, the AR content may show the user changes in fluid flow. This may provide the user with guidance on moving in one direction or another. For example, a surgical hub may receive indicators of an intended action on a target area. The indicators may include images captured via the AR device showing a surgical instrument positioned on or near the target area. The indicators may include images captured via a surgical scope showing a surgical instrument positioned on or near the target area. The surgical hub may obtain predicted outcomes associated with performing an intended action on a target area and may include these predicted outcomes in the AR content. The predicted outcomes may be determined based on visual data received from the surgical scope and surgical data received from the surgical instrument. The predicted outcomes may be determined by the surgical hub or with the help of a remote server.

[0011] AR devices may, for example, provide an auditory overlay in an operating room (OR), not as a substitute for sound, but in addition to hearing sound. The AR system can communicate specific information only to targeted individuals within the OR who can access that information.

[0012] Further embodiments of the present invention provide the following examples. 1. A surgical hub, A communication array configured to be operably connected to at least one display, multiple AR (augmented reality) devices, and at least one surgical instrument, A processor, and the processor, To obtain AR control parameters, Obtaining a first data stream from at least one surgical instrument for display on a first display of at least one display, Based on AR control parameters, determine a first AR content to be overlaid on a first data stream displayed on a first display via a first AR device among multiple AR devices, Based on AR control parameters, determine a second AR content to be overlaid on a first data stream displayed on a first display via a second AR device among multiple AR devices, wherein the second AR content is different from the first AR content. A surgical hub configured to transmit a first AR content and a second AR content to a first AR device and a second AR device, respectively.

[0013] For example, in Example 1, the first display may be a primary display or a secondary display. Obtaining AR control parameters may include determining, measuring, or receiving AR control parameters. The processor may determine the first and / or second AR content based on whether the AR control parameters exceed a threshold.

[0014] For example, in Example 1, the processor may be further configured to transmit a first data stream to a first display. The processor may be further configured to overlay first AR content on the first data stream via a first AR device and to overlay second AR content on the first data stream via a second AR device.

[0015] For example, in Example 1, the first and / or second AR device may be safety glasses, augmented reality goggles, or a head-mounted display having an augmented reality display. Surgical instruments may include laparoscopic instruments, endoscopic instruments, laparoscopes, electrosurgical instruments, ultrasonic surgical instruments, and / or surgical stapling instruments. The first data stream may be, or may include, video images of the surgical site in the patient. The first display may include, or may include, at least one of the following: an instrument display mounted on a surgical instrument such as at least one surgical instrument, a display in or within the operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, or a wrist-mounted display.

[0016] For example, in Example 1, the surgical hub may include an input device configured to receive inputs and transmit them to a processor. The input device may include an interactive display. The first display may include an input device. The input device may allow AR control parameters, user roles or intended users for the first and / or second AR devices, and / or display control indicators to be input and transmitted to the processor.

[0017] 2. AR control parameters include user roles, and the processor, Identifying a first user role associated with a first AR device, Based on the first user role, determine the first overlay dataset to be included in the first AR content, Identifying a second user role associated with a second AR device, The surgical hub of Example 1 is further configured to determine, based on a second user role, a second overlay dataset to be included in a second AR content, wherein the second overlay dataset is different from the first overlay dataset.

[0018] For example, in Example 2, identifying the first user role associated with the first AR device may include identifying, selecting, or receiving data that identifies the intended user role for the first AR device. Identifying the second user role associated with the second AR device may include identifying, selecting, or receiving data that identifies the intended user role for the second AR device. The user and user role may include a surgeon, a surgeon's assistant, and / or a medical professional who may be located inside or outside the sterile field. Based on the first user role, determining the first overlay data set may include generating a first overlay data set for inclusion in the first AR content of the data intended for display to the intended user role of the first AR device. Based on the second user role, determining the second overlay data set may include generating a second overlay data set for inclusion in the second AR content of the data intended for display to the intended user role of the second AR device.

[0019] 3. The processor is further configured to instruct the first AR device to: detect that a user of the first AR device is viewing the first display; and in response to the detection, overlay the first AR content on the first data stream displayed on the first display via the first AR device. The surgical hub of Example 1 or 2.

[0020] 4. The first AR content includes at least one of: steps for the use of a surgical instrument, device settings, device status, device instructions for use, operating parameters, or indicators of detected abnormalities. The surgical hub of any one of Examples 1 to 3.

[0021] 5. The surgical hub of any one of Examples 1 to 4, wherein the first AR content includes at least one of preoperative imaging, intraoperative imaging, instrument data, or treatment instructions.

[0022] For example, in any one of Examples 1 to 5, the first and / or second AR content is a process for the use of a surgical instrument, device settings, device status, device instructions for use, operating parameters, indicators of detected abnormalities, preoperative imaging, intraoperative imaging, instrument data, treatment instructions, indicators of predicted results when the user-intended action is executed, changes in the flow of body fluids, blood flow within blood vessels such as low blood flow or high blood flow, indicators of which blood vessels supply blood to a tumor, tumor marking, tumor margin, resection margin, resection or tumor margin adjusted during a surgical procedure, real-time Doppler monitoring, preoperative MRI data, indocyanine green (ICG) fluorescence imaging data, identification information of the user operating a surgical tool, real-time surgical data received from another connected system, proximity alerts when a surgical instrument such as at least one surgical instrument (or its distal tip) moves within a specific range of an important structure, a severed portion of a blood vessel to represent a blood supply affected by a cutting path, a portion of an anatomical structure that can become non-viable due to lack of blood or air due to a cutting path, a hub-proposed cutting path or a processor-proposed cutting path, indicators of expected problems, visualization of obscured portions of the surgical site, landmarks shown in an image of the surgical site, and / or anatomical identification information generated based on preoperative images, and may include at least one of them.

[0023] 6. The surgical hub of any one of Examples 1 to 5, wherein the AR control parameters include at least one of the orientation of the user with respect to the first display, the progress of the surgical procedure, the surgical context, real-time user input, or preconfigured user preferences.

[0024] For example, in any one of Examples 1 to 6, the AR control parameters may include at least one of the following: user orientation to the first display, user orientation to the first display, user head orientation to the first display, whether the user is looking at the first display, progress of the surgical procedure, surgical context, real-time user input, user roles associated with the first and / or second AR devices, or intended user roles for the first and / or second AR devices, display type of the first display, and / or pre-configured user preferences.

[0025] 7. The first AR content to be overlaid on the first data stream displayed on the first display is further determined based on the display type associated with the first display, and the processor, One of the surgical hubs, Examples 1-6, is further configured to determine a third AR content to be overlaid on content displayed on the second display via a first AR device, based on the display type associated with the second display of at least one of the displays.

[0026] For example, in any one of Examples 1 to 7, the first AR content to be overlaid on a first data stream displayed on a first display may be further determined based on the display type of the first display, and the processor is further configured to determine a third AR content to be overlaid on content displayed on a second display via a first AR device, based on the display type of a second display among at least one of the displays.

[0027] 8. Display type is Instrument displays placed on smart surgical instruments, or A shared display in the operating room, or A surgical hub, example 7, including at least one of the following: a personal display.

[0028] For example, in any one of Examples 1 to 8, the display type of the first and / or second display may include an instrument display mounted on a surgical instrument such as at least one surgical instrument, a display in or within an operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, or a wrist-worn display.

[0029] 9. The processor, Receiving display control indicators from the first AR device, A surgical hub, one of Examples 1-8, is further configured to adjust content for display on a first display based on received display control indicators.

[0030] The display control indicator may be a selection of display content. The surgical hub may include an input device configured to receive and transmit the display control indicator to a processor. The input device may include an interactive display. The first display may include an input device.

[0031] 10. The processor, The first AR device detects whether it is outside the boundary of the surgical room, One of the surgical hubs in Examples 1-9 is further configured to disable the transmission of the first AR content to the first AR device in response to detecting that the first AR device is outside the boundary of the surgical room.

[0032] 11. The processor, Receiving indicators of intended actions in the target area, Obtaining predictive results associated with performing intended actions on the target region, One of the surgical hubs, Examples 1-10, is further configured to include the prediction results in the first AR content.

[0033] For example, in Example 11, obtaining a prediction result may include determining the prediction result. The intended action on the target area may include at least one of the following: a surgical procedure performed on the target tissue, a procedure for using a surgical instrument, a clamping operation, a cutting operation, a stapling operation, a transverse incision operation, and / or the application of ultrasound and / or RF energy.

[0034] The indicator may include images acquired via a surgical scope such as a laparoscope or endoscope and / or surgical instrument and / or a first AR device, showing that the surgical instrument is positioned on or near a target area, visual input from a camera in the operating room, and sensor input from a surgical device such as a surgical instrument.

[0035] The prediction results may be determined based on visual data received from a surgical scope and / or surgical data received from at least one surgical instrument, such as a surgical instrument. The prediction results may be obtained and / or determined by a processor in the surgical hub, or by a remote server.

[0036] The surgical hub may be equipped with a situational awareness system, and the processor may use the situational awareness system to infer indicators of intended actions on the target area.

[0037] 12. A surgical hub of Example 11, in which the indicator includes a visualization captured via a first AR device or surgical scope, the visualization indicating that a surgical instrument is positioned on or near a target area.

[0038] 13. The processor, A surgical hub, example 11 or 12, further configured to determine a predictive outcome based on visual data received from a surgical scope and surgical data received from at least one surgical instrument.

[0039] For example, in Example 13, at least one surgical instrument may include a surgical scope. Obtaining or determining a prediction result may include determining the prediction result based on visual data received from the surgical scope and surgical data received from at least one surgical instrument.

[0040] 14. The processor, Acquire visual data from a surgical scope and sensor input data from at least one surgical instrument, The process involves sending visual data and sensor input data to a remote server, A surgical hub, example 11 or 12, further configured to receive prediction results from a remote server.

[0041] For example, in any one of Examples 11-14, receiving an index may include obtaining visual data from a surgical scope and / or sensor input data from at least one surgical instrument, transmitting the visual data and sensor input data to a remote server, and receiving prediction results from the remote server.

[0042] 15. A surgical hub, one of Examples 1-14, in which the first AR content includes at least one of visual or audible content.

[0043] For example, in any one of Examples 11 to 15, the first AR content includes at least one of visual content, audible content, and / or haptic content.

[0044] 16. A surgical hub, A communication array configured to be operably connected to at least one display, an AR (augmented reality) device, and at least one surgical instrument, A processor, and the processor, Obtaining a data stream from at least one surgical instrument for display on at least one display, To generate AR content to overlay on the display, Determining whether the user of the AR device is looking at the display, A surgical hub configured to detect when a user of an AR device is looking at the display, and to overlay AR content onto the content displayed on the display via the AR device.

[0045] For example, in Example 16, the processor may be further configured to send a data stream to a display and / or display the data stream on the display. Overlaying AR content on content displayed on the display via an AR device may include overlaying AR content on a data stream displayed on the display via an AR device.

[0046] For example, in Example 16, the AR device may be safety glasses, augmented reality goggles, or a head-mounted display having an augmented reality display. Surgical instruments may include laparoscopic instruments, endoscopic instruments, laparoscopes, electrosurgical instruments, ultrasonic surgical instruments, and / or surgical stapling instruments. The data stream may be, or may include, video images of the surgical site in the patient. The display may include, or may include, an instrument display mounted on a surgical instrument such as at least one surgical instrument, a display in or within the operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, or a wrist-mounted display.

[0047] For example, in Example 16, the surgical hub may include an input device configured to receive inputs and transmit them to a processor. The input device may include an interactive display. The first display may include an input device. The input device may allow AR control parameters, user roles, and / or display control indicators to be input and transmitted to the processor.

[0048] 17. The processor, Determining whether the user of the first AR device is looking at the first display, A surgical hub, one of Examples 1-15, is further configured to detect that a user of the first AR device is looking at the first display, and to overlay the first AR content onto the content displayed on the first display via the first AR device.

[0049] For example, in Example 17, overlaying the first AR content onto content displayed on the first display via the first AR device may include overlaying the first AR content onto a first data stream displayed on the first display via the first AR device.

[0050] 18. The display is the first of at least one display, and the processor is To generate a second AR content to be overlaid on a second display of at least one display, wherein the second AR content is different from the AR content to be overlaid on the first display. Determining whether the user of the AR device is looking at the second display, A surgical hub, example 16 or 17, is further configured to, upon detecting that the user of the AR device is looking at a second display, overlay second AR content onto the content displayed on the second display via the AR device.

[0051] For example, in Example 17, the first display may be a primary display. The second display may be a secondary display. The second display may include, or may not include, an instrument display placed on a surgical instrument such as at least one surgical instrument, an operating room or in-operating room display, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, or a wrist-worn display.

[0052] 19. The processor is further configured to stop supplying AR content associated with the display via the AR device when it detects that the user of the AR device is no longer looking at the display, in any one of the surgical hubs in Examples 16-18.

[0053] For example, in any one of Examples 16-19, the processor may be further configured to stop overlaying AR content on content displayed on the display via the AR device when it detects that the user of the AR device is no longer looking at the display.

[0054] 20. The processor, Receiving display control indicators from an AR device, A surgical hub, one of the examples 16-19, is further configured to adjust the content to be displayed on the display based on the received display control indicators.

[0055] The display control indicator may be a selection of display content. The surgical hub may include an input device configured to receive and transmit the display control indicator to a processor. The input device may include an interactive display. The display may include an input device.

[0056] 21. The processor is further configured to identify the user role associated with the AR device, and the AR content is generated based on the user role, as described in any one of Examples 16-20 of the Surgical Hub.

[0057] Identifying user roles associated with an AR device may include identifying, selecting, or receiving data that identifies the intended user role for a second AR device. AR content may be generated based on the user role by using or including data intended for display to the intended user role of the AR device. Users and user roles may include surgeons, surgical assistants, and / or medical professionals who may be located inside or outside a sterile field. [Brief explanation of the drawing]

[0058] [Figure 1] This is a block diagram of a computer-assisted interactive surgical system. [Figure 2] This shows an exemplary surgical system used for performing surgical procedures in the operating room. [Figure 3]This disclosure illustrates, in at least one aspect, a visualization system, a robotic system, and an exemplary surgical hub paired with an intelligent instrument. [Figure 4] The present disclosure, in at least one aspect, describes a surgical data network having a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] An exemplary computer-implemented interactive surgical system is shown. [Figure 6] An exemplary surgical setup is shown, featuring multiple modules connected to a modular control tower. [Figure 7] An example of a surgical instrument or tool is shown. [Figure 8] This shows a surgical instrument or tool that has a motor that can be activated to perform various functions. [Figure 9] This is a diagram illustrating an exemplary situational awareness surgical system. [Figure 10] This shows an exemplary timeline of a surgical procedure and reasoning that can be created by a surgical hub from data detected at each step of the surgical procedure. [Figure 11] This is a block diagram of a computer-implemented interactive surgical system. [Figure 12] This illustrates the functional architecture of an exemplary computer-implemented interactive surgical system. [Figure 13] This exhibits an exemplary computer-implemented interactive surgical system configured to adaptively generate control program updates for modular devices. [Figure 14] An exemplary surgical system is shown, comprising a handle having a controller and a motor, an adapter releasably connected to the handle, and a loading unit releasably connected to the adapter. [Figure 15A] This section describes an exemplary flow for determining the operating mode and for operating in the determined mode. [Figure 15B] This shows an exemplary flow for changing the operating mode. [Figure 16] This shows the primary display of a surgical hub equipped with global and local displays. [Figure 17] An example of a primary display for a surgical hub is shown. [Figure 18] The diagram depicts a surgeon using surgical instruments, including a handle assembly housing and a wireless circuit board, during a surgical procedure. The surgeon is wearing a set of safety glasses. [Figure 19] This is a diagram illustrating an exemplary operating room (OR) setup. [Figure 20] This is a block diagram of a gesture recognition system. [Figure 21] This illustrates exemplary role-based interaction and control related to augmented reality and deviceless control systems. [Figure 22] This exhibits exemplary procedure-based interactions and controls related to augmented reality and deviceless control systems. [Figure 23] This is a schematic diagram illustrating an exemplary visualization of anatomical structures using a spectral surgical visualization system. [Figure 24] This is a diagram of a surgical instrument access route for a video-assisted thoracoscopic surgery (VATS) procedure, according to at least one aspect of the present disclosure. [Figure 25] This is a diagram of various coordinate systems related to VATS treatment according to at least one aspect of the present disclosure. [Figure 26] This figure illustrates exemplary changes in display orientation and user control in response to changes in the orientation of surgical instruments. [Figure 27] Depicts an exemplary camera view of a surgical procedure. [Figure 28] An illustrative representation of a surgical visualization system according to at least one aspect of this disclosure is shown. [Figure 29]This shows an exemplary model of an anatomical structure generated by an exemplary surgical visualization system. [Figure 30] An exemplary representation of an exemplary model is shown according to at least one aspect of this disclosure. [Figure 31] This shows an exemplary display of an exemplary model of an anatomical structure generated by an exemplary surgical visualization system. [Figure 32] This is a diagram of an exemplary fused image generated from a multispectral EMR source. [Figure 33] This illustrates exemplary action steps and progressions that can be detected by the system's exemplary situational awareness capabilities. [Figure 34A] This example shows a series of surgical procedures involving multi-image analysis of the surgical site. [Figure 34B] This example shows a series of surgical procedures involving multi-image analysis of the surgical site. [Figure 34C] This example shows a series of surgical procedures involving multi-image analysis of the surgical site. [Figure 35] An example of an augmented video image of a preoperative video image is shown, which is enhanced using data that identifies the displayed elements. [Figure 36] This example shows an augmented reality overlay on a target region using preoperative tumor data and real-time Doppler monitoring. [Figure 37] This shows an exemplary flow for a hub operating under a hierarchical visualization control mode. [Figure 38] This shows an exemplary flow for a hub operating under a hierarchical visualization control mode. [Figure 39] This provides a detailed, illustrative flow for a hub operating under a visualization control mode where the secondary display is an augmented reality (AR) device. [Figure 40] This shows an exemplary flow for a hub operating under a visualization control mode that supports situational awareness. [Figure 41] This shows an exemplary flow for a hub operating under a visualization control mode that supports situational awareness. [Figure 42] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports adjusting the display based on adjusted display events. [Figure 43] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports AR capabilities. [Figure 44] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports AR capabilities. [Figure 45] This illustrates an exemplary flow of a hub operating under a visualization control mode that supports role-based AR capabilities. [Figure 46] This illustrates an exemplary flow of a hub operating under a visualization control mode with AR capabilities that support overlays on various displays. [Modes for carrying out the invention]

[0059] The applicant of this application also owns the following U.S. patent applications filed concurrently, the contents of which are incorporated herein by reference. U.S. Patent Application No. 16 / 209,416, titled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS," filed on December 4, 2018. U.S. Patent Application No. 15 / 940,671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER" (Agent Reference Number END8502USNP), filed on March 29, 2018. U.S. Patent Application No. 16 / 182,269, filed on November 6, 2018, entitled "IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE" (Agent Reference Number END9018USNP3), U.S. Patent Application No. 16 / 729,747, titled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS" (Agent Reference Number: END9217USNP1), filed on December 31, 2019. U.S. Patent Application No. 16 / 729,778, filed on December 31, 2019, entitled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE" (Agent Reference Number END9219USNP1), U.S. Patent Application No. 16 / 729,807, titled "METHOD OF USING IMAGING DEVICES IN SURGERY" (Agent Reference Number: END9228USNP1), filed on December 31, 2019. U.S. Patent Application No. 15 / 940,654, entitled "SURGICAL HUB SITUATIONAL AWARENESS," filed on March 29, 2018 (Agent Reference Number END8501USNP), U.S. Patent Application No. 15 / 940,671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER" (Agent Reference Number END8502USNP), filed on March 29, 2018. U.S. Patent Application No. 15 / 940,704, filed on March 29, 2018, entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT" (Agent Reference Number END8504USNP), U.S. Patent Application No. 16 / 182,290, filed on November 6, 2018, entitled "SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION" (Agent Reference Number END9018USNP5), • U.S. Patent No. 9,011,427, titled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued on April 21, 2015. U.S. Patent No. 9,123,155, issued on September 1, 2015, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES" • U.S. Patent Application No. 16 / 209,478, filed on December 4, 2018, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE" (Agent Reference Number END9015USNP1), and U.S. Patent Application No. 16 / 182,246, entitled "ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES" (Agent Reference Number END9016USNP1), filed on November 6, 2018.

[0060] The surgical hub may have collaborative interaction with one of several means of displaying images from a laparoscope and information from one of several other smart devices. The hub may have the ability to interact with these multiple displays using algorithms or control programs that enable combined display and control of data distributed across displays communicating with the hub.

[0061] Referring to Figure 1, the computer-implemented interactive surgical system 100 may include one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 which may include a remote server 113 connected to a storage device 105). Each surgical system 102 may include at least one surgical hub 106 that communicates with the cloud 104 which may include the remote server 113. In one example, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some embodiments, the surgical system 102 may include M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P are integers of 1 or more.

[0062] In various embodiments, the visualization system 108 may include one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described in the section “Advanced Imaging Acquisition Module” in the United States. U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” is incorporated herein by reference in its entirety.

[0063] As shown in Figure 2, the primary display 119 is positioned in the sterile field so that it is visible to the operator on the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may include a first non-sterile display 107 and a second non-sterile display 109, facing opposite directions from each other. The visualization system 108, guided by the hub 106, is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 can cause the visualization system 108 to display snapshots of the surgical site recorded by the imaging device 124 on the non-sterile displays 107 or 109 while maintaining live video of the surgical site on the primary display 119. The snapshots on the non-sterile displays 107 or 109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0064] In one embodiment, the hub 106 is also configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a primary display 119 in the sterile field, which can then be viewed by a sterile operator at the operating table. In one example, the input may take the form of modifications to a snapshot displayed on the non-sterile display 107 or 109, which can then be sent to the primary display 119 by the hub 106.

[0065] Referring to Figure 2, the surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 is also configured to coordinate the flow of information to the display of the surgical instrument 112. For example, U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," is incorporated herein by reference in its entirety. Diagnostic inputs or feedback entered by a non-sterile operator in the visualization tower 111 can be sent by the hub 106 to the surgical instrument display 115 in the sterile field, which can then be viewed by the operator of the surgical instrument 112. Examples of surgical instruments suitable for use with the surgical system 102 are described, for example, in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the entire disclosure of which is incorporated herein by reference.

[0066] Figure 2 depicts an example of a surgical system 102 used to perform a surgical procedure on a patient lying on an operating table 114 in a surgical operating room 116. A robotic system 110 may be used as part of the surgical system 102 in a surgical procedure. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. While the surgeon views the surgical site through the surgeon's console 118, the patient-side cart 120 can manipulate at least one detachably connected surgical tool 117 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 124, which can be manipulated by the patient-side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 118.

[0067] Other types of robotic systems can be readily adapted for use with surgical system 102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2018-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed on 4 December 2019, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the entire disclosure of which is incorporated herein by reference.

[0068] Various examples of cloud-based analytical methods implemented by Cloud104 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed on 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the entire disclosure of which is incorporated herein by reference.

[0069] In various embodiments, the imaging device 124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0070] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0071] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0072] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0073] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0074] The imaging device may employ multispectral monitoring to distinguish between topology and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in the section "Advanced Imaging Acquisition Module" in S. U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the entire disclosure of which is incorporated herein by reference. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is ready for surgical treatment. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and restraints within that area.

[0075] Referring here to Figure 3, a hub 106 is depicted that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Valuable time can be lost dealing with this problem during surgical procedures. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The modular enclosure 136 of the hub provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data contacts and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from the remote surgical site to a suction and irrigation module slidably received within the hub enclosure. In one embodiment, the hub enclosure comprises a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house various generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and second data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a modular enclosure 136 of a hub that enables modular integration of a generator module 140, a smoke exhaust module 126, and a suction / irrigation module 128. The modular enclosure 136 of the hub further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the modular enclosure 136 of the hub. The modular enclosure 136 of the hub may be configured to facilitate the insertion of multiple generators and interactive communication between generators docked to the modular enclosure 136 of the hub, so that multiple generators function as a single generator.

[0076] Figure 4 shows a surgical data network 201 comprising a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud-based system (e.g., a cloud 204 which may include a remote server 213 connected to a storage device 205). In one embodiment, the modular communication hub 203 comprises a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be connected to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, allowing data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network includes additional mechanisms that allow traffic to pass through a monitored surgical data network, comprising each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0077] Modular devices 1a-1n located in the operating room may be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and operation. Data associated with devices 1a-1n may also be transferred to a local computer system 210 for local data processing and operation. Modular devices 2a-2m located in the same operating room may also be connected to a network switch 209. The network switch 209 may be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to a cloud 204. Data associated with devices 2a-2n may be transferred to a cloud 204 via the network router 211 for data processing and operation. Data associated with devices 2a-2m may also be transferred to a local computer system 210 for local data processing and operation.

[0078] It will be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accommodate multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, the devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module.

[0079] In one embodiment, the surgical data network 201 may include a combination of a network hub, network switches, and network routers connecting devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” can be used as a metaphor for “Internet,” but the term is not limited in that way. Thus, the term “cloud computing” can be used herein to refer to “a type of internet-based computing,” in which various services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 203 and / or computer system 210 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 203 and / or computer system 210. The cloud infrastructure may be maintained by a cloud service provider. In this context, a cloud service provider may be an entity that coordinates the use and control of one or more devices 1a-1n / 2a-2m located within the operating room. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located within the operating room. The hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0080] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and increased patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures and to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204 or the local computer system 210, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be carried out for tissue-specific sites and conditions. Such data analysis may also involve further prognostic analysis, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0081] The operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of the devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a-1n located within the same operating room network. The network hub 207 can collect data in packet form and transmit them to the router in half-duplex mode. The network hub 207 does not store any media access control / Internet protocol (MAC / Internet Protocol, IP) for transferring device data. Only one of the devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 does not need to have a routing table or intelligence regarding the destination of the information and broadcasts all network data to each connection and to the remote server 213 (Figure 4) on the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and can cause bottlenecks.

[0082] Operating room devices 2a-2m may be connected to network switch 209 via a wired or wireless channel. Network switch 209 operates within the data link layer of the OSI model. Network switch 209 may be a multicast device for connecting devices 2a-2m located in the same operating room to a network. Network switch 209 can transmit data in frame form to network router 211 and can operate in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0083] The network hub 207 and / or network switch 209 may be connected to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates a route for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, such as different networks located in different operating rooms of the same medical facility or different operating rooms of different medical facilities. The network router 211 can transmit data in packet form to the cloud 204 and can operate in full-duplex mode. Multiple devices can transmit data simultaneously. The network router 211 uses IP addresses to transfer data.

[0084] In one example, the network hub 207 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0085] In this example, the operating room devices 1a-1n / 2a-2m can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with the modular communication hub 203 via the Bluetooth wireless technology standard to build a personal area network (PAN). The operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via a number of wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G and beyond. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0086] The modular communication hub 203 can function as a central connection point for one or all of the operating room devices 1a-1n / 2a-2m and can handle a data type known as a frame. Frames can carry data generated by devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, it is amplified and transmitted to the network router 211, which then transfers this data to cloud computing resources using a number of wireless or wired communication standards or protocols as described herein.

[0087] The modular communication hub 203 may be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room equipment 1a~1n / 2a~2m.

[0088] Figure 5 shows a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204 which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to multiple operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 connected to a computer system 210.

[0089] As shown in the example in Figure 5, the modular control tower 236 may be connected to an imaging module 238 which may be connected to an endoscope 239, a generator module 240 which may be connected to an energy device 241, a smoke exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 optionally connected to a display 237, and a non-contact sensor module 242. The operating room equipment may be connected to cloud computing resources and data storage via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the device / instrument 235 and the visualization system 208 may be connected to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be connected to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with images and superimposed images.

[0090] Figure 6 shows a surgical hub 206 comprising multiple modules connected to a modular control tower 236. The modular control tower 236 may comprise a modular communication hub 203, such as a network connectivity device, and a computer system 210, for example, local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 203, and data associated with the modules may be transferred to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and local displays 217. Communication to the cloud 204 can be done via either a wired communication channel or a wireless communication channel.

[0091] The surgical hub 206 can use a non-contact sensor module 242 to measure the dimensions of the surgical field and generate a map of the operating room using either an ultrasonic non-contact measuring device or a laser non-contact measuring device. The ultrasonic-based non-contact sensor module can scan the operating room by transmitting bursts of ultrasound and receiving echoes as they bounce off the surrounding walls of the operating room. This is described in the section "Surgical Hub Spatial Awareness Within an Operating Room" in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the entire disclosure of which is incorporated herein by reference, in which the sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. A laser-based non-contact sensor module can, for example, scan an operating room by transmitting laser light pulses, receive the laser light pulses reflected from the outer walls of the operating room, compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room, and adjust the Bluetooth pairing distance limit.

[0092] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 can be connected via a system bus to a communication module 247, storage 248, memory 249, non-volatile memory 250, and an input / output interface 251. The system bus may be any of several types of bus structures, including a memory bus or memory controller, peripheral bus or external bus, and / or local bus, using any various available bus architectures, including but not limited to 9-bit buses, Industrial Standard Architecture (ISA), Microchannel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0093] Processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0094] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0095] System memory can include volatile and non-volatile memory. The basic input / output system (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory can include random-access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0096] The computer system 210 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Disk storage may include, but is not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage may include the above-mentioned storage media independently or in combination with other storage media. Other storage media may include, but are not limited to, optical disk drives such as compact disc ROM devices (CD-ROMs), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0097] It should be understood that the computer system 210 may include software that acts as an intermediary between the user and basic computer resources, as described in a preferred operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, can function to control and allocate the resources of the computer system. System applications can leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0098] The user can input commands or information to the computer system 210 via input devices connected to the I / O interface 251. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to the output device. Output adapters may be provided to indicate the existence of several output devices, particularly monitors, displays, speakers, and printers, which require special adapters. Examples of output adapters include, but are not limited to, video and sound cards that provide means of connection between the output device and the system bus. It should be noted that other devices and / or systems of devices, such as remote computers, can provide both input and output capabilities.

[0099] Computer system 210 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces can encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include Fiber Distributed Data Interfaces (FDDI), Copper Distributed Data Interfaces (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated services digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0100] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5 and 6, and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction, multiple data (SIMD) or multiple-instruction, multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0101] The communication connection section can refer to the hardware / software used to connect a network interface to a bus. For the sake of clarity of the example, the communication connection section is shown as being inside the computer system, but it may also be outside the computer system 210. For illustrative purposes only, the hardware / software required for connecting to a network interface may include internal and external technologies such as modems including standard telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0102] Figure 7 shows a logic diagram of a control system 470 for a surgical instrument or tool according to one or more embodiments of the present disclosure. The system 470 may include a control circuit. The control circuit may include a microcontroller 461 having a processor 462 and memory 468. For example, one or more of sensors 472, 474, and 476 provide real-time feedback to the processor 462. A motor 482 driven by a motor driver 492 drives an I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 480 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to a processor 462 which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control the firing of the I-beam, the movement of the occluder, the rotation of the shaft, and joint movement. The display 473 can display various operating conditions of the instrument and may include a touchscreen function for data input. The information displayed on the display 473 can be overlaid with images acquired via the endoscopic imaging module.

[0103] In one embodiment, the microcontroller 461 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 461 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels.

[0104] In one embodiment, the microcontroller 461 may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0105] The microcontroller 461 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 461 may include a processor 462 and memory 468. The electric motor 482 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 492 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 480 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety.

[0106] The microcontroller 461 may be programmed to provide precise control over the velocity and position of the displacement member and joint motion system. The microcontroller 461 may be configured to calculate the response within its software. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response may be a well-adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and this can detect external influences on the system.

[0107] In some examples, the motor 482 may be controlled by a motor driver 492 and may be used by a surgical instrument or tool launching system. In various forms, the motor 482 may be, for example, a brushed DC-driven motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 482 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 492 may comprise, for example, an H-bridge driver including a field-effect transistor (FET). The motor 482 may be powered by a power supply assembly removably mounted on a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for powering a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be a lithium-ion battery that can be coupled to and detached from the power supply assembly.

[0108] The motor driver 492 may be the A3941, available from Allegro Microsystems, Inc. The A3941 492 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET), particularly designed for inductive loads such as brushed DC motors. The driver 492 may include a proprietary charge pump regulator, which can provide full (>10V) gate drive to battery voltages down to 7V, and can enable the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET can be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be easily substituted for use in the tracking system 480 with an absolute positioning system.

[0109] The tracking system 480 may include a controlled motor drive circuit arrangement comprising a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for the absolute positioning system can provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element which may be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 472 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include a contact-type displacement sensor or a non-contact-type displacement sensor.A linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0110] The electric motor 482 may include a rotary shaft that operably interfaces with a gear assembly mounted on a displacement member by meshing with a set of drive teeth or a rack. The sensor element may be operably connected to the gear assembly such that one rotation of the position sensor 472 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply can provide power to the absolute positioning system, and an output indicator can display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing with the corresponding drive gear of the gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0111] One rotation of the sensor element associated with the position sensor 472 may correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor mechanism may be connected via a gear reduction that results in the position sensor 472 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 472 can complete multiple rotations relative to the full stroke of the displacement member.

[0112] To provide a unique position signal for two or more rotations of the position sensor 472, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches may be fed back to the microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 of the sensor mechanism may include an array of analog rotation sensors such as a magnetic sensor or potentiometer, or an array of analog Hall effect elements, which output a unique combination of position signals or values.

[0113] The position sensor 472 may comprise any number of magnetic sensing elements, such as magnetic sensors, which are classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors can include numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0114] In one embodiment, the position sensor 472 of a tracking system 480 equipped with an absolute positioning system may be equipped with a magnetic rotation absolute positioning system. The position sensor 472 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 472 interfaces with a microcontroller 461 to provide an absolute positioning system. The position sensor 472 may be a low-voltage, low-power component and may include four Hall effect elements in the area of ​​the position sensor 472 located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement a simple and efficient algorithm for computing hyperbolic and trigonometric functions that require only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information can be transmitted to the microcontroller 461 via a standard serial communication interface such as a serial peripheral interface (SPI). The position sensor 472 can provide 12-bit or 14-bit resolution. The position sensor 472 may also be an AS5055 chip, which is available in a small QFN 16-pin 4x4x0.85mm package.

[0115] The tracking system 480, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of the physical system will be by knowing the input, the calculated response of the physical system may take into account properties such as mass, inertia, viscous friction, and inductive resistance.

[0116] The absolute positioning system can provide the absolute position of a displacement member when the device is powered on, without requiring the displacement member to be moved back or forward to a reset (zero or home) position, as may be necessary with conventional rotary encoders that simply count the number of forward or backward strokes taken by the motor 482 to estimate the position of the device actuator, drive bar, knife, etc.

[0117] For example, a sensor 474, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of an end effector, such as the amplitude of strain exerted on the anvil during clamping, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to a processor 462. Instead of, or in addition to, sensor 474, a sensor 476, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, sensor 476, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and deform into contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by a firing bar. Alternatively, a current sensor 478 may be used to measure the current drawn in by the motor 482. The force required to propel the launching member forward may correspond, for example, to the current drawn in by the motor 482. The measured force can be converted into a digital signal and provided to the processor 462.

[0118] In one embodiment, a strain gauge sensor 474 can be used to measure the force applied to tissue by the end effector. A strain gauge can be connected to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 474, such as a micro-strain gauge configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 474 can measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 462 of the microcontroller 461. A load sensor 476 can measure the force used to operate a knife element, for example, to cut tissue trapped between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor can also be converted into a digital signal and provided to the processor 462.

[0119] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 474 and 476 respectively, can be used by the microcontroller 461 to characterize the selected position of the launcher and / or the corresponding values ​​of the launcher's velocity. In one example, memory 468 can store techniques, equations, and / or lookup tables that can be used by the microcontroller 461 during evaluation.

[0120] The control system 470 for surgical instruments or tools may also include a wired or wireless communication circuit for communicating with a modular communication hub 203, as shown in Figures 5 and 6.

[0121] Figure 8 shows a surgical instrument or tool equipped with multiple motors that can be activated to perform various functions. In a particular example, the first motor can be activated to perform the first function, the second motor can be activated to perform the second function, the third motor can be activated to perform the third function, the fourth motor can be activated to perform the fourth function, and so on. In a particular example, the multiple motors of the robotic surgical instrument 600 can be activated individually to produce firing, closing, and / or jointing motions in the end effector. The firing, closing, and / or jointing motions can be transmitted to the end effector, for example, via a shaft assembly.

[0122] In certain examples, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably connected to a firing motor drive assembly 604, which can be configured to transmit the firing motion generated by the motor 602 to an end effector, specifically to displace an I-beam element. In certain examples, the firing motion generated by the motor 602 may, for example, deploy a staple from a staple cartridge into tissue captured by the end effector and / or advance the cutting edge of the I-beam element to cut the captured tissue. The I-beam element can be retracted by reversing the direction of the motor 602.

[0123] In certain examples, the surgical instrument or tool may include a closure motor 603. The closure motor 603 may be operably coupled to a closure motor drive assembly 605, which may be configured to specifically displace a closure tube to close the anvil and transmit the closure motion generated by the motor 603 to an end effector to compress tissue between the anvil and the staple cartridge. The closure motion allows the end effector to transition from an open configuration to an approach configuration, for example, to capture tissue. The end effector may be moved to an open position by reversing the direction of the motor 603.

[0124] In certain examples, a surgical instrument or tool may include, for example, one or more articular motors 606a, 606b. The motors 606a, 606b may be operably connected to corresponding articular motor drive assemblies 608a, 608b, which may be configured to transmit the articular motion generated by the motors 606a, 606b to an end effector. In certain examples, the articular motion may cause, for example, the end effector to articulate relative to the shaft.

[0125] As described herein, a surgical instrument or tool may include multiple motors that can be configured to perform various independent functions. In certain examples, multiple motors of a surgical instrument or tool can be activated individually or separately to perform one or more functions while other motors remain stopped. For example, articulation motors 606a and 606b can be activated to articulate an end effector while firing motor 602 remains stopped. Alternatively, firing motor 602 can be activated to fire multiple staples and / or advance a cutting edge while articulation motor 606 remains stopped. Furthermore, a closure motor 603 may be activated simultaneously with firing motor 602 to advance the closure tube and I-beam element distally, as described in more detail below herein.

[0126] In certain examples, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In certain examples, the common control module 610 may correspond to one of the multiple motors at a time. For example, the common control module 610 may be individually connectable and disconnectable to multiple motors of the robotic surgical instrument. In certain examples, multiple motors of the surgical instrument or tool may share one or more common control modules, such as the common control module 610. In certain examples, multiple motors of the surgical instrument or tool can engage with the common control module 610 individually and selectively. In certain examples, the common control module 610 can selectively switch between interfacing with one of the multiple motors of the surgical instrument or tool and interfacing with another of the multiple motors of the surgical instrument or tool.

[0127] In at least one example, the common control module 610 can be selectively switched between an operable engagement with the articulation motors 606a, 606b and an operable engagement with either the firing motor 602 or the closing motor 603. In at least one example, as shown in Figure 8, the switch 614 can move or transition between multiple positions and / or states. For example, in the first position 616, the switch 614 may electrically connect the common control module 610 to the firing motor 602; in the second position 617, the switch 614 may electrically connect the common control module 610 to the closing motor 603; in the third position 618a, for example, the switch 614 may electrically connect the common control module 610 to the first articulation motor 606a; and in the fourth position 618b, the switch 614 may electrically connect the common control module 610 to the second articulation motor 606b. In certain examples, a separate common control module 610 may also be electrically connected to the launch motor 602, the closing motor 603, and the joint motion motors 606a, 606b. In certain examples, the switch 614 may be a mechanical switch, an electromechanical switch, a solid switch, or any preferred switching mechanism.

[0128] Each of the motors 602, 603, 606a, and 606b may be equipped with a torque sensor for measuring the output torque on the motor shaft. The force on the end effector may be sensed in any conventional manner, such as by force sensors on the outside of the jaws or by torque sensors on the motors that actuate the jaws.

[0129] In various examples, as shown in Figure 8, the common control module 610 may include a motor driver 626 which may comprise one or more H-bridge FETs. The motor driver 626 may modulate the power transmitted from the power supply 628 to the motor connected to the common control module 610 based, for example, on input from a microcontroller 620 ("controller"). In certain examples, as described herein, the microcontroller 620 can be used, for example, to determine the current drawn by the motor while the motor is connected to the common control module 610.

[0130] In certain examples, the microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-temporary computer-readable media or memory units 624 ("memory"). In certain examples, the memory 624 may store various program instructions, which, when executed, can cause the processor 622 to perform some of the functions and / or calculations described herein. In certain examples, one or more of the memory units 624 may be linked to the processor 622, for example.

[0131] In certain examples, power supply 628 may be used to supply power to, for example, a microcontroller 620. In certain examples, power supply 628 may include a battery (or "battery pack" or "power pack") such as a lithium-ion battery. In certain examples, the battery pack may be configured to be removably attached to a handle in order to supply power to a surgical instrument 600. A number of battery cells connected in series may be used as power supply 628. In certain examples, power supply 628 may be, for example, replaceable and / or rechargeable.

[0132] In various examples, the processor 622 can control the motor driver 626 to control the position, direction of rotation, and / or speed of a motor connected to a common control module 610. In specific examples, the processor 622 can signal the motor driver 626 to stop and / or disable a motor connected to a common control module 610. The term “processor,” as used herein, should be understood to include any suitable microprocessor, microcontroller, or other basic computing device that integrates the functions of a computer’s central processing unit (CPU) on one or up to several integrated circuits. A processor can be a multipurpose programmable device that accepts digital data as input, processes that data according to instructions stored in memory, and provides the results as output. It may be an example of sequential digital logic, as it has internal memory. A processor can operate with numbers and symbols represented in binary.

[0133] The processor 622 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In a particular example, the microcontroller 620 may be, for example, the LM 4F230H5QR available from Texas Instruments. In at least one embodiment, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes, among other features readily available in the product datasheet, 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and one or more 12-bit ADCs with 12 analog input channels. Other microcontrollers may be readily substituted for use with module 4410. Therefore, this disclosure should not be limited to this context.

[0134] Memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600, which can be connected to a common control module 610. For example, memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the joint movement motors 606a, 606b. Such program instructions can cause the processor 622 to control the firing function, closing function, and joint movement function according to input from an algorithm or control program of the surgical instrument or tool.

[0135] For example, one or more mechanisms and / or sensors, such as sensor 630, can be used to alert the processor 622 to program instructions that should be used in a particular setting. For example, sensor 630 can alert the processor 622 to use program instructions related to the firing, closing, and joint movement of the end effector. In a particular example, sensor 630 may include a position sensor that can be used to sense the position of switch 614, for example. Thus, if the processor 622 detects, for example, via sensor 630 that switch 614 is in a first position 616, it can use a program instruction associated with the firing of the end effector's I-beam; if the processor 622 detects, for example, via sensor 630 that switch 614 is in a second position 617, it can use a program instruction associated with the closing of the anvil; and if the processor 622 detects, for example, via sensor 630 that switch 614 is in a third position 618a or a fourth position 618b, it can use a program instruction associated with the joint movement of the end effector.

[0136] Figure 9 shows a diagram of a context-aware surgical system 5100 relating to at least one aspect of the present disclosure. In some examples, the data source 5126 may include, for example, a modular device 5102 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 5122 (e.g., an EMR database containing patient records), and a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor). The surgical hub 5104 may be configured to derive contextual information about a surgical procedure from the data, for example, based on a particular combination of received data or a particular order in which data is received from the data source 5126. Contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function relating to some aspects of the surgical hub 5104 for deriving or inferring information about a surgical procedure from received data may also be referred to as “context-aware.” In one example, the surgical hub 5104 may incorporate a context-aware system, which is hardware and / or programming associated with the surgical hub 5104, that derives contextual information related to the surgical procedure from the received data.

[0137] The situation awareness system of the surgical hub 5104 can be configured to derive contextual information from data received from the data source 5126 in various different ways. For example, the situation awareness system may include a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 5122, patient monitoring device 5124, and / or modular device 5102) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In this example, the situation awareness system may include a lookup table that stores pre-characterized contextual information about the surgical procedure, associated with one or more inputs (or ranges of inputs) corresponding to that contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situation awareness system to control the modular device 5102. In the example, contextual information received by the situation awareness system of the surgical hub 5104 may be associated with a specific control adjustment of one or more modular devices 5102, or a set of control adjustments. Alternatively, the situation awareness system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments of one or more modular devices 5102 when contextual information is provided as input.

[0138] The surgical hub 5104, which incorporates a situational awareness system, can bring many advantages to the surgical system 5100. One advantage may include improved interpretation of detected and collected data, which improves processing accuracy during the course of the surgical procedure and / or the use of the data. Returning to the previous example, the situational awareness surgical hub 5104 can determine what type of tissue is being operated on, so if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the type of tissue.

[0139] The type of tissue being operated on can affect the adjustments made to the compression rate and load threshold of surgical staple fasteners and cutting instruments for measuring specific interstitial gaps. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic or abdominal surgery, thereby allowing the surgical hub 5104 to determine whether the tissue being clamped by the end effector of the surgical staple fastener and cutting instrument is the lung (in the case of thoracic surgery) or the stomach (in the case of abdominal surgery). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical staple fastener and cutting instrument according to the type of tissue.

[0140] The type of body cavity being operated on during an aeration procedure can affect the function of the smoke exhauster. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing aeration) and determine the type of procedure. Generally, since a certain type of procedure may be performed in a specific body cavity, the surgical hub 5104 can appropriately control the motor speed of the smoke exhauster to match the body cavity being operated on. Thus, the situational awareness surgical hub 5104 can provide a consistent amount of smoke exhaust for both thoracic and abdominal surgeries.

[0141] The type of procedure being performed can affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, arthroscopy may require a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopy. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situational awareness surgical hub 5104 can determine what type of surgical procedure is being performed and then customize the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue shape for the surgical procedure. Furthermore, the situation-aware surgical hub 5104 can be configured to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument not simply for each procedure, but throughout the course of the surgical procedure. The situation-aware surgical hub 5104 can determine which step of the surgical procedure is being performed or is continuing, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type according to the steps of the surgical procedure.

[0142] In the example, the surgical hub 5104 may also derive data from an additional data source 5126 to improve conclusions drawn from one data source 5126. The contextually aware surgical hub 5104 may enhance data received from the modular device 5102 with contextual information constructed from other data sources 5126 regarding the surgical procedure. For example, the contextually aware surgical hub 5104 may be configured to determine whether hemostasis has been achieved (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, video or image data may not be conclusive. Therefore, in one example, the surgical hub 5104 may be further configured to make a determination regarding the integrity of the staple line or tissue weld by comparing physiological measurements (e.g., blood pressure detected by a BP monitor communicably connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably connected to the surgical hub 5104). In other words, the context-aware system of the surgical hub 5104 can provide additional context when analyzing visualization data by considering physiological measurement data. This additional context can be useful when the visualization data itself may not be definitive or may be incomplete.

[0143] For example, the situational awareness surgical hub 5104 can proactively activate the generator to which the RF electrosurgical instrument is connected if it determines that the instrument will be needed in a subsequent step of the procedure. By proactively activating the energy source, it is possible to make the instrument ready for use as soon as the preceding step of the procedure is completed.

[0144] The situational awareness surgical hub 5104 can determine whether the current or subsequent steps of a surgical procedure require different views or magnifications on the display, according to the characteristics of the surgical site that the surgeon is expected to need to see. The surgical hub 5104 can then appropriately and proactively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108), thereby automatically adjusting the display throughout the surgical procedure.

[0145] The situational awareness surgical hub 5104 can determine which step of a surgical procedure is being performed or will be performed next, and whether specific data or comparisons of data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0146] Errors can be checked during the setup of a surgical procedure or during the procedure itself. For example, the situation-aware surgical hub 5104 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 may be configured to compare a list of items for the procedure and / or a list of devices paired with the surgical hub 5104 to a recommended or expected manifest of items and / or devices for a given surgical procedure. If discrepancies exist between the lists, the surgical hub 5104 can be configured to provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, and / or other surgical articles are missing. In some examples, the surgical hub 5104 may be configured to determine the relative distance or relative position of the modular device 5102 and the patient monitoring device 5124, for example, by proximity sensors. The surgical hub 5104 can compare the relative position of the devices to a recommended or predicted layout for a particular surgical procedure. If a discontinuity exists between the layouts, the surgical hub 5104 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout.

[0147] The situational awareness surgical hub 5104 can determine whether a surgeon (or other healthcare professional) is making an error or deviating from a set of actions expected during the course of a surgical procedure. For example, the surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the course of the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub 5104 has determined to be performed. In some examples, the surgical hub 5104 can be configured to provide alerts indicating that an unexpected action is being taken or an unexpected device is being used at a particular step in the surgical procedure.

[0148] Surgical instruments (and other modular devices 5102) may be adjusted for the specific context of each surgical procedure (e.g., adjustments for different tissue types) and may verify actions during the surgical procedure. The following steps, data, and display adjustments may be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the procedure.

[0149] Figure 10 shows a timeline 5200 illustrating an exemplary surgical procedure and contextual information that the surgical hub 5104 can derive from data received from data source 5126 at each stage of the surgical procedure. Refer to Figure 9 for the following description of the timeline 5200 shown in Figure 9. The timeline 5200 may depict the typical steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy, beginning with the setup of the operating room and ending with the transfer of the patient to the postoperative recovery room. The contextually aware surgical hub 5104 can receive data from data source 5126 throughout the surgical procedure, including data generated each time a healthcare professional uses the modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular device 5102 and other data sources 5126, and can continuously derive estimations (i.e., contextual information) about the procedure in progress as new data is received, such as which step of the procedure is being performed at any given time. The contextual awareness system of the surgical hub 5104 can, for example, record data about the procedure to generate a report, verify the steps being taken by the healthcare professional, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and perform any other such actions as described herein.

[0150] As the first step 5202 in this exemplary procedure, hospital staff can retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery. In the second step 5204, staff can scan incoming medical supplies for the procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies that may be used in various types of procedures to confirm that the combination of supplies matches a thoracic procedure. Furthermore, the surgical hub 5104 may also determine that the procedure is not a wedge resection (because the incoming supplies either do not include specific supplies required for a thoracic wedge resection or are otherwise not compatible with a thoracic wedge resection). In the third step 5206, healthcare workers can scan the patient band via a scanner 5128 that is communicably connected to the surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data. In the fourth step 5208, a medical professional turns on the auxiliary device. The auxiliary devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they may include a smoke exhauster, an air insulator, and a medical imaging device. Once activated, the auxiliary device, which is a modular device 5102, can automatically pair with a surgical hub 5104, which may be located within a specific vicinity of the modular device 5102, as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 that it is paired with during this pre-operative or initialization phase. In this particular example, the surgical hub 5104 can determine that the surgical procedure is a VATS procedure based on this specific combination of paired modular devices 5102. Based on the combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team.Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing. In the fifth step 5210, personnel attach the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 may be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, it can confirm that the patient is in the operating room, for example, as described in process 5207. In the sixth step 5212, healthcare personnel may induce anesthesia in the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from the modular device 5102 and / or the patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 5212 is completed, the preoperative portion of the lung segmentectomy is complete and the surgical portion begins.

[0151] In the seventh step 5214, the lung of the patient being operated on may collapse (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer, for example, that the patient's lung has collapsed from the ventilator data. The surgical hub 5104 can compare the detection of the patient's lung collapse with the anticipated steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has begun and determine that collapsing the lung may be the first surgical step in this particular procedure. In the eighth step 5216, a medical imaging device 5108 (e.g., a scope) may be inserted and video footage from the medical imaging device may be started. The surgical hub 5104 may receive medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has begun. Furthermore, the surgical hub 5104 can determine that a particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resections have not been taken into consideration by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Contextual information regarding the type of procedure being performed can be determined in various ways using data from the medical imaging device 124 (Figure 2), for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structures, by monitoring the number or type of medical imaging device being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used. For example, one technique for performing a VATS lobectomy may position the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while one technique for performing a VATS segmentectomy may position the camera in an anterior intercostal position relative to the segmental fissure. A situational awareness system can be trained, for example, using pattern recognition or machine learning techniques to recognize the position of a medical imaging device according to the visualization of a patient's anatomical structure.An exemplary technique for performing VATS lobectomy can utilize a single medical imaging device. An exemplary technique for performing VATS segmentectomy utilizes multiple cameras. An exemplary technique for performing VATS segmentectomy utilizes an infrared light source (which can be communicably connected to a surgical hub as part of a visualization system) to visualize the segmental fissure, but this is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0152] In the ninth step 5218, the surgical team may begin the incision step of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the steps of the procedure described above) corresponds to the incision step. In the tenth step 5220, the surgical team may proceed to the ligation step of the procedure. The surgical hub 5104 can receive data from surgical stapling and cutting instruments indicating that an instrument is being emitted, and can therefore infer that the surgeon is ligating arteries and veins. As in the previous step, the surgical hub 5104 can derive this inference by cross-referencing the received data from the surgical stapling and cutting instruments with the steps in the read-out process. In the eleventh step 5222, the segmental resection portion of the procedure may be performed. The surgical hub 5104 can infer that a surgeon is transversely incising parenchymal tissue based on data from surgical stapling and cutting instruments (including data from their cartridges). The cartridge data may correspond, for example, to the size or type of staples being fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples being fired is used for parenchymal tissue (or other similar tissue types), thereby allowing the surgical hub 5104 to infer that the segmental resection portion of the procedure is being performed. Subsequently, in the twelfth step 5224, the nodule incision step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby allowing the surgical hub 5104 to make this inference.It should be noted that different instruments are better suited to specific tasks, so surgeons should periodically alternate between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the specific stage of the procedure. Therefore, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which stage of the procedure the surgeon is performing. Once the 12th step 5224 is completed, the incision is closed, and the postoperative portion of the procedure can begin.

[0153] In the 13th step 5226, the patient can be de-anesthetized. The surgical hub 5104 may estimate that the patient is waking from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase). Finally, the 14th step 5228 may be that the medical personnel remove the various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 may estimate that the patient is being transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. As can be seen from this exemplary procedure description, the surgical hub 5104 can determine or estimate when each step of a given surgical procedure is being performed, according to the data received from various data sources 5126 that are communicably connected to the surgical hub 5104.

[0154] As shown in the first step 5202 of the timeline 5200 depicted in Figure 10, in addition to using patient data from the EMR database to estimate the type of surgical procedure to be performed, the patient data can also be used by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0155] Figure 11 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system may be configured to monitor and analyze data relating to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analytics system. The cloud-based analytics system may be described as a surgical system, but is not necessarily limited to that, and may be a cloud-based medical system. As shown in Figure 11, the cloud-based analytics system may include a plurality of surgical instruments 7012 (which may be the same as or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same as or similar to hub 106), and a surgical data network 7001 (which may be the same as or similar to network 201) for connecting the surgical hubs 7006 to a cloud 7004 (which may be the same as or similar to cloud 204). Each of the plurality of surgical hubs 7006 may be communicably connected to one or more surgical instruments 7012. The hub 7006 may also be communicably connected to a cloud 7004 of a computer-implemented interactive surgical system via a network 7001. The cloud 7004 may be a remote, centralized source of hardware and software for storing, manipulating, and communicating data generated based on the operation of various surgical systems. As shown in Figure 11, access to the cloud 7004 may be achieved via the network 7001, which may be the Internet or another suitable computer network. The surgical hub 7006, which can be connected to the cloud 7004, can be considered the client side of a cloud computing system (i.e., a cloud-based analytics system). Surgical instruments 7012 may be paired with the surgical hub 7006 for the control and execution of various surgical procedures or actions described herein.

[0156] In addition, the surgical instrument 7012 may be equipped with transceivers for data transmission to and from the corresponding surgical hub 7006 (which may also be equipped with transceivers). The combination of the surgical instrument 7012 and the corresponding hub 7006 can indicate a specific location, such as an operating room within a medical facility (e.g., a hospital) for providing medical surgery. For example, the memory of the surgical hub 7006 can store location data. As shown in Figure 11, the cloud 7004 comprises a central server 7013 (which may be the same as or similar to the remote server 7013), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7006. The central server 7013 of the cloud 7004 collectively manages the cloud computing system, which includes monitoring requests from the client module 7006 and managing the processing power of the cloud 7004 to perform those requests. Each central server 7013 may comprise one or more processors 7008 connected to a suitable memory device 7010, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 7010 may include machine-executable instructions that, when executed, cause the processor 7008 to run a data analysis module 7034 for cloud-based data analysis, operation, recommendations, and other operations described below. Furthermore, the processor 7008 may run the data analysis module 7034 independently or in conjunction with a hub application running independently by the hub 7006. The central server 7013 may also include a database 2212 of aggregated medical data that may reside in memory 2210.

[0157] Based on connections to various surgical hubs 7006 via network 7001, cloud 7004 can aggregate data from various surgical instruments 7012 and specific data generated by their corresponding hubs 7006. Such aggregated data may be stored in the aggregated medical database 7012 of cloud 7004. Specifically, cloud 7004 can advantageously perform data analysis and operations on the aggregated data to derive insights and / or perform functions that individual hubs 7006 cannot achieve on their own. For this purpose, as shown in Figure 11, cloud 7004 and surgical hubs 7006 are communicated together to send and receive information. The I / O interface 7006 is connected to multiple surgical hubs 7006 via network 7001. In this way, the I / O interface 7006 can be configured to transfer information between the surgical hubs 7006 and the aggregated medical data database 7011. Thus, the I / O interface 7006 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hub 7006. These requests may be sent to the hub 7006 via the hub application. The I / O interface 7006 may include one or more high-speed data ports, including a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 7004 to the hub 7006. The hub application server 7002 of the cloud 7004 may be configured to host and supply sharing capabilities to a software application (e.g., a hub application) running on the surgical hub 7006. For example, the hub application server 7002 may manage requests from the hub application through the hub 7006, control access to the aggregated medical data database 7011, and perform load balancing. The data analysis module 7034 will be described in more detail with reference to Figure 12.

[0158] The configurations of specific cloud computing systems described in this disclosure may be designed to address a variety of problems arising in the context of medical surgeries and procedures performed using medical devices such as surgical instruments 7012, 112. Specifically, surgical instrument 7012 may be a digital surgical device configured to interact with the cloud 7004 in order to implement techniques for improving surgical outcomes. Various surgical instruments 7012 and / or surgical hubs 7006 may include a touch-controlled user interface so that a clinician can control the manner of interaction between the surgical instruments 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0159] Figure 12 is a block diagram showing the functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analytics system may include a number of data analytics modules 7034 that can be executed by a processor 7008 of the cloud 7004 to provide data analytics solutions to problems that arise particularly in the medical field. As shown in Figure 12, the functionality of the cloud-based data analytics module 7034 may be supported via a hub application 7014 hosted by a hub application server 7002 that can be accessed on a surgical hub 7006. The cloud processor 7008 and the hub application 7014 may work together to execute the data analytics module 7034. An application program interface (API) 7016 can define a set of protocols and routines corresponding to the hub application 7014. In addition, the API 7016 can manage the storage and retrieval of data to a medical database 7012 aggregated for the operation of the application 7014. A cache 7018 may also store data (e.g., temporarily) and be linked to the API 7016 for more efficient retrieval of data used by the application 7014. The data analysis module 7034 in Figure 12 may include modules for resource optimization 7020, data collection and aggregation 7022, authorization and security 7024, control program updates 7026, patient outcome analysis 7028, recommendations 7030, and data classification and prioritization 7032. Other suitable data analysis modules may also be implemented by the cloud 7004 in several embodiments. In one embodiment, the data analysis module may be used for specific recommendations based on the analysis of trends, outcomes, and other data.

[0160] For example, the data collection and aggregation module 7022 may be used to generate self-describing data (e.g., metadata), which may include identifying prominent features or configurations (e.g., trends), managing redundant datasets, and storing data in paired datasets that may be grouped by surgery but not necessarily linked to the actual surgery date and surgeon. Specifically, paired datasets generated from the operation of surgical instruments 7012 may involve applying a binary classification, such as bleeding or non-bleeding events. More generally, the binary classification may be characterized as either a desirable event (e.g., a successful surgical procedure) or an undesirable event (e.g., a surgical instrument 7012 misfired or misused). The aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Thus, the data collection and aggregation module 7022 can generate aggregated metadata or other organized data based on the raw data received from the surgical hub 7006. For this purpose, the processor 7008 can be operationally linked to the hub application 7014 and the aggregated medical data database 7011 in order to execute the data analysis module 7034. The data acquisition and aggregation module 7022 may store the aggregated and organized data in the aggregated medical data database 2212.

[0161] The resource optimization module 7020 can be configured to analyze this aggregated data to determine the optimal use of resources for a particular healthcare facility or group of healthcare facilities. For example, the resource optimization module 7020 can determine the optimal reorder point for surgical staple fasteners 7012 for a group of healthcare facilities based on the corresponding predicted demand for those staple fasteners 7012. The resource optimization module 7020 can also evaluate the resource use or other operational configurations of various healthcare facilities to determine whether resource use can be improved. Similarly, the recommendation module 7030 can be configured to analyze the aggregated data from the data collection and aggregation module 7022 to provide recommendations. For example, the recommendation module 7030 can recommend to a healthcare facility (e.g., a healthcare service provider such as a hospital) that a particular surgical instrument 7012 should be upgraded to an improved version based, for example, on the error rate being higher than expected. In addition, the recommendation module 7030 and / or the resource optimization module 7020 can recommend better supply chain parameters, such as product reorder points, and provide suggestions for different surgical instruments 7012, their use, or procedures to improve surgical outcomes. Medical facilities can receive such recommendations via the corresponding surgical hub 7006. More specific recommendations regarding the parameters or configurations of various surgical instruments 7012 can also be provided. The hub 7006 and / or the surgical instrument 7012 may each have a display screen that shows data or recommendations provided by the cloud 7004.

[0162] The patient outcome analysis module 7028 can analyze surgical outcomes associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 may also analyze and evaluate other potential operating parameters. In this regard, the recommendation module 7030 may use these other potential operating parameters to make recommendations based on the resulting better surgical outcomes, such as better sealing or less bleeding. For example, the suggestion module 7030 may send a suggestion to the surgical instrument 7006 regarding the use of a particular cartridge with the corresponding stapled surgical instrument 7012. Thus, the cloud-based analytics system may be configured to analyze large amounts of collected raw data while controlling for common variables and to provide centralized recommendations across multiple healthcare facilities (favorably determined based on aggregated data). For example, the cloud-based analytics system can analyze, evaluate, and / or aggregate data in ways that a single healthcare facility cannot analyze independently, such as the type of medical procedure, the type of patient, the number of patients, and geographical similarities among healthcare providers using similar types of instruments. The control program update module 7026 can be configured to implement various surgical instrument 7012 recommendations when the corresponding control program is updated. For example, the patient outcome analysis module 7028 can identify correlations linking specific control parameters to successful (or unsuccessful) outcomes. Such correlations may be addressed when the updated control program is transmitted to the surgical instrument 7012 via the control program update module 7026. Updates to the instrument 7012, which may be transmitted via the corresponding hub 7006, may incorporate aggregated outcome data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. In addition, the patient outcome analysis module 7028 and the recommendation module 7030 can identify improved ways of using the instrument 7012 based on the aggregated outcome data.

[0163] The cloud-based analytics system may include security features implemented by Cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other preferred security credentials. These credentials may be stored in memory 7010 and associated with permitted cloud access levels. For example, based on providing accurate credentials, a surgical hub 7006 may be granted access to communicate with the cloud to a predetermined extent (e.g., to send or receive certain defined types of information). For this purpose, the aggregated medical data database 7011 of Cloud 7004 may include a database of certified credentials to verify the accuracy of the provided credentials. Different credentials may be associated with various levels of authorization for interaction with Cloud 7004, such as a predetermined access level for receiving data analysis generated by Cloud 7004. Furthermore, for security purposes, the cloud may maintain databases of hubs 7006, instruments 7012, and other devices, which may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be permitted to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be prevented from fully functioning when paired with the corresponding hub 7006. Additionally or alternatively, the cloud 7004 may flag instruments 7012 based on non-conformity or other specified criteria. In this way, counterfeit medical devices and the improper reuse of such devices across the entire cloud-based analysis system can be identified and addressed.

[0164] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal that may represent, for example, authorization credentials for access to the corresponding hub 7006 and the cloud 7004. A wired transceiver can also be used to transmit a signal. Such authorization credentials can be stored in the respective memory devices of the surgical instrument 7012. The authorization and security module 7024 can determine whether the authorization credentials are accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credentials may also be encrypted, for example, by using hash-based encryption. Upon transmitting appropriate authorization, the surgical instrument 7012 may signal to the corresponding hub 7006 and ultimately the cloud 7004 to indicate that the instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may transition to a state in which it can receive medical data for storage in the aggregated medical data database 7011. This readiness for data transmission can be indicated, for example, by an optical indicator on the instrument 7012. Cloud 7004 can also send signals to surgical instruments 7012 to update their associated control programs. Cloud 7004 can send signals directed to specific categories of surgical instruments 7012 (e.g., electrosurgical instruments) to ensure that software updates for control programs are sent only to the appropriate surgical instruments 7012. Furthermore, Cloud 7004 may be used to implement system-wide solutions to address local or global issues based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, Cloud 7004 may modify the authorization credentials corresponding to this group to implement operational lockout for that group.

[0165] A cloud-based analytics system can enable monitoring of multiple healthcare facilities (e.g., healthcare facilities such as hospitals) to determine improved practices and recommend changes accordingly (e.g., via the suggestion module 2030). Thus, the processor 7008 of Cloud 7004 can analyze data associated with individual healthcare facilities to identify facilities and aggregate that data with other data associated with other healthcare facilities. Groups may be defined, for example, based on similar operational practices or geographical location. In this way, Cloud 7004 can provide a wide range of analytics and recommendations to healthcare facility groups. The cloud-based analytics system can also be used for enhanced contextual awareness. For example, the processor 7008 may predictively model the effect of cost and effectiveness recommendations for a particular facility (compared to overall operations and / or various medical procedures). The cost and effectiveness associated with that particular facility can also be compared to the corresponding local areas of other facilities or any other equivalent facilities.

[0166] The data classification and prioritization module 7032 may prioritize and classify data based on severity (e.g., the severity, unexpectedness, or suspiciousness of the medical event associated with the data). This classification and prioritization may be used in conjunction with the functions of other data analysis modules 7034 described herein to improve the cloud-based analysis and operations described herein. For example, the data classification and prioritization module 7032 may assign priorities to data analyses performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels may result in specific responses from the cloud 7004 (corresponding to the level of urgency), such as increased priority for rapid response, special processing, exclusion from the aggregated medical data database 7011, or other preferred responses. Furthermore, if necessary, the cloud 7004 may send requests (e.g., push messages) via the hub application server for additional data from the corresponding surgical instrument 7012. Push messages may result in notifications displayed on the corresponding hub 7006 to request support or additional data. This push message may be needed in situations where the cloud detects a significant anomaly or outlier and the cloud is unable to determine the cause of that anomaly. The central server 7013 can be programmed to trigger this push message in certain critical situations, such as when data is determined to be different from expected values ​​by exceeding a predetermined threshold, or when security is suspected to be involved.

[0167] Further illustrative details regarding the various functions described are provided in the following descriptions. Each of the descriptions may utilize a cloud architecture, as shown in Figures 11 and 12 as examples of hardware and software implementations.

[0168] Figure 13 shows a block diagram of a computer-implemented adaptive surgical system 9060 configured to adaptively generate control program updates for modular devices 9050, according to at least one aspect of the present disclosure. In some examples, the surgical system may include a surgical hub 9000, a plurality of modular devices 9050 communicably connected to the surgical hub 9000, and an analysis system 9100 communicably connected to the surgical hub 9000. While a single surgical hub 9000 may be depicted, it should be noted that the surgical system 9060 may include any number of surgical hubs 9000 that can be connected to form a network of surgical hubs 9000 communicably connected to the analysis system 9010. In some examples, the surgical hub 9000 may include a processor 9010 connected to memory 9020 for executing stored instructions and a data relay interface 9030 to which data is transmitted to the analysis system 9100. In some examples, the surgical hub 9000 may further include a user interface 9090 having an input device 9092 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 9094 (e.g., a display screen) for providing output to the user. The output may include data from queries entered by the user, suggestions for products or mixtures of products for use in a given procedure, and / or instructions for actions to be taken before, during, or after a surgical procedure. The surgical hub 9000 may further include an interface 9040 for communicatively connecting modular devices 9050 to the surgical hub 9000. In one embodiment, the interface 9040 may include a transceiver that can be communicatively connected to the modular devices 9050 via a wireless communication protocol. The modular devices 9050 may include, for example, surgical stapling and cutting instruments, electrosurgical instruments, ultrasound instruments, inflators, ventilators, and display screens. In some examples, the surgical hub 9000 can be further communicated to one or more patient monitoring devices 9052, such as an EKG monitor or a BP monitor.In some examples, the surgical hub 9000 can be further communicated to one or more databases 9054 or external computer systems, such as the EMR database of the medical facility where the surgical hub 9000 is located.

[0169] When the modular device 9050 is connected to the surgical hub 9000, the surgical hub 9000 can sense or receive perioperative data from the modular device 9050 and then associate the received perioperative data with surgical procedure outcome data. The perioperative data can indicate how the modular device 9050 was controlled during the surgical procedure. The procedure outcome data includes data associated with the outcome from the surgical procedure (or the process thereof), which may include whether the surgical procedure (or the process thereof) had a positive or negative outcome. For example, outcome data may include whether the patient suffered a postoperative complication from a particular procedure, or whether there was leakage (e.g., bleeding or air leakage) at the staple line or incision line. The surgical hub 9000 can obtain the surgical procedure outcome by receiving data from an external source (e.g., from the EMR database 9054), by directly detecting the outcome (e.g., via one of the connected modular devices 9050), or by inferring the occurrence of the outcome through a situational awareness system. For example, data on postoperative complications can be retrieved from the EMR database 9054, and data on staple line or incision line leakage can be directly detected or inferred by the situational awareness system. Surgical procedure result data can be inferred by the situational awareness system from data received from various data sources, including the modular device 9050 itself, the patient monitoring device 9052, and the database 9054 to which the surgical hub 9000 is connected.

[0170] The surgical hub 9000 can transmit data and result data from associated modular devices 9050 to the analysis system 9100 for processing on the analysis system 9100. By transmitting both perioperative data indicating how the modular devices 9050 are controlled and procedure result data, the analysis system 9100 can correlate different modes of control of the modular devices 9050 with surgical outcomes for specific procedure types. In some examples, the analysis system 9100 may include a network of analysis servers 9070 configured to receive data from the surgical hub 9000. Each of the analysis servers 9070 may include memory and a memory-coupled processor that executes instructions stored therein to analyze the received data. In some examples, the analysis servers 9070 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 9100 can then learn the optimal or preferred operating parameters for various types of modular devices 9050, generate adjustments to the control programs of the modular devices 9050 in the field, and then transmit (or "push") updates to the control programs of the modular devices 9050.

[0171] Further details regarding the computer-implemented interactive surgical system 9060, including the surgical hub 9000 and various modular devices 9050 that can be connected thereto, will be described in relation to Figures 5 and 6.

[0172] Figure 14 provides a surgical system 6500 according to the present disclosure, which may include a surgical instrument 6502 capable of communicating with a console 6522 or portable device 6526 via a wired or wireless connection through a local area network 6518 or a cloud network 6520. In various embodiments, the console 6522 and portable device 6526 may be any suitable computing device. The surgical instrument 6502 may include a handle 6504, an adapter 6508, and a loading unit 6514. The adapter 6508 is releasably coupled to the handle 6504, and the loading unit 6514 is releasably coupled to the adapter 6508 so that the adapter 6508 transmits force from the drive shaft to the loading unit 6514. The adapter 6508 or the loading unit 6514 may include force gauges (not explicitly shown) placed therein for measuring the force exerted on the loading unit 6514. The loading unit 6514 may include an end effector 6530 comprising a first jaw 6532 and a second jaw 6534. The loading unit 6514 may also be a multi-firing loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without the loading unit 6514 having to be removed from the surgical site to reload the loading unit 6514.

[0173] The first jaw 6532 and the second jaw 6534 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 6532 may be configured to include a replaceable multi-fired fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) which may be configured to fire at least one fastener multiple times, or which may be fired two or more times before being replaced. The second jaw 6534 may include an anvil that deforms the fasteners or otherwise secures them around the tissue as they are ejected from the multi-fired fastener cartridge.

[0174] The handle 6504 may include a motor connected to the drive shaft so as to affect the rotation of the drive shaft. The handle 6504 may include a control interface for selectively starting the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces, which can be used by a clinician to start the motor.

[0175] The control interface of the handle 6504 communicates with the controller 6528 of the handle 6504 to selectively start the motor and affect the rotation of the drive shaft. The controller 6528 may be located within the handle 6504 and is configured to receive input from the control interface and adapter data from the adapter 6508 or loading unit data from the loading unit 6514. The controller 6528 may analyze the input from the control interface and the data received from the adapter 6508 and / or loading unit 6514 to selectively start the motor. The handle 6504 may also include a display that can be viewed by a clinician while the handle 6504 is in use. The display may be configured to show portions of the adapter or loading unit data before, during, or after firing the instrument 6502.

[0176] The adapter 6508 may include an adapter identification device 6510 located therein, and the loading unit 6514 may include a loading unit identification device 6516 located therein. The adapter identification device 6510 may communicate with the controller 6528, and the loading unit identification device 6516 may communicate with the controller 6528. It will be understood that the loading unit identification device 6516 may communicate with the adapter identification device 6510 which relays or bypasses communication from the loading unit identification device 6516 to the controller 6528.

[0177] The adapter 6508 may also include a plurality of sensors 6512 (one shown) positioned around it to detect various states of the adapter 6508 or the environment (e.g., when the adapter 6508 is connected to a loading unit, when the adapter 6508 is connected to a handle, when the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside the adapter 6508, the number of times the adapter 6508 has fired, the peak force of the adapter 6508 during firing, the total amount of force applied to the adapter 6508, the peak recoil force of the adapter 6508, the number of pauses of the adapter 6508 during firing, etc.). The plurality of sensors 6512 can provide input to the adapter identification device 6510 in the form of data signals. The data signals from the plurality of sensors 6512 may be stored in the adapter identification device 6510 or used to update adapter data stored in the adapter identification device 6510. The data signals from the plurality of sensors 6512 may be analog or digital. Multiple sensors 6512 may include force gauges for measuring the force exerted on the loading unit 6514 during firing.

[0178] The handle 6504 and adapter 6508 can be configured to interconnect the adapter identification device 6510 and the loading unit identification device 6516 with the controller 6528 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., electrical contacts including electrical contacts that engage with each other to transmit energy and signals). Additionally or alternatively, the electrical interface may be a non-contact electrical interface that transmits energy and signals wirelessly (e.g., inductively). It is also intended that the adapter identification device 6510 and the controller 6528 may be able to wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0179] The handle 6504 may include a transmitter 6506 configured to transmit instrument data from the controller 6528 to other components of the system 6500 (e.g., LAN 6518, cloud 6520, console 6522, or portable device 6526). The transmitter 6506 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 6500. For example, the controller 6528 may transmit instrument data to the console 6528 including the serial number of an attached adapter (e.g., adapter 6508) attached to the handle 6504, the serial number of a loading unit (e.g., loading unit 6514) attached to the adapter, and the serial number of a multi-fire fastener cartridge (e.g., multi-fire fastener cartridge) loaded in the loading unit. The console 6522 may then send back data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, to the controller 6528. The controller 6528 can display a message on the local device display, or send a message via the transmitter 6506 to the console 6522 or portable device 6526, which can then display the message on the display 6524 or the portable device screen, respectively.

[0180] Figure 15A shows an exemplary flow for determining the operating mode and operating in the determined mode. Computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems may be configured to be updatable. Such updates may include the inclusion of features and benefits that were not available to the user before the update. These updates can be established by any method of hardware, firmware, and software updates suitable for introducing features to the user. For example, computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems can be updated using interchangeable / replaceable (e.g., hot-swappable) hardware components, flashable firmware devices, and updatable software systems.

[0181] Renewal can be conditional on any appropriate criteria or set of criteria. For example, renewal can be conditional on one or more hardware capabilities of the system, such as processing power, bandwidth, or resolution. For example, renewal can be conditional on one or more software aspects, such as the purchase of a certain software code. For example, renewal can be conditional on a purchased service tier. A service tier may represent the characteristics and / or set of characteristics that a user is entitled to use in connection with a computer-implemented interactive surgical system. A service tier may be determined by a license code, e-commerce server authentication interaction, hardware key, username / password combination, biometric interaction, public / private key exchange interaction, etc.

[0182] In 10704, system / device parameters may be identified. System / device parameters may be any element or set of elements that make an update conditional. For example, a computer-implemented interactive surgical system may detect a specific bandwidth for communication between a modular device and a surgical hub. For example, a computer-implemented interactive surgical system may detect instructions for purchasing a specific service tier.

[0183] In 10708, the operating mode may be determined based on identified system / device parameters. This determination may be made by a process that maps system / device parameters to operating modes. The process may be manual and / or automated. The process may be the result of local and / or remote calculations. For example, client / server interaction can be used to determine the operating mode based on identified system / device parameters. For example, local software and / or locally embedded firmware may be used to determine the operating mode based on identified system / device parameters. For example, a hardware key, such as a secure microprocessor, can be used to determine the operating mode based on identified system / device parameters.

[0184] In 10710, operation may proceed according to a determined operating mode. For example, a system or device may proceed to operate in a default operating mode. For example, a system or device may proceed to operate in an alternative operating mode. The operating mode may be indicated by control hardware, firmware, and / or software already present in the system or device. The operating mode may be indicated by newly installed / updated control hardware, firmware, and / or software.

[0185] Figure 15B shows an exemplary functional block diagram for changing the operating mode. The upgradeable element 10714 may include an initialization component 10716. The initialization component 10716 may include any hardware, firmware, and / or software suitable for determining the operating mode. For example, the initialization component 10716 may be part of a system or device startup procedure. The initialization component 10716 may be involved in interactions for determining the operating mode of the upgradeable element 10714. For example, the initialization component 10716 may interact with, for example, a user 10730, an external resource 10732, and / or a local resource 10718. For example, the initialization component 10716 may receive a license key from user 10730 to determine the operating mode. The initialization component 10716 may query an external resource 10732, such as a server, using the serial number of the upgradeable device 10714 to determine the operating mode. For example, the initialization component 10716 may query the local resource 10718, such as a local query to determine the amount of available bandwidth and / or a local query for a hardware key, to determine the operating mode.

[0186] An upgradeable element 10714 may include one or more operational components 10720, 10722, 10726, 10728, and an operational pointer 10724. The initialization component 10716 can instruct the operational pointer 10724 to direct the operation of the upgradeable element 10741 to the operational components 10720, 10722, 10726, 10728 corresponding to the determined operational mode. The initialization component 10716 may also instruct the operational pointer 10724 to direct the operation of the upgradeable element to the default operational component 10720. For example, the default operational component 10720 may be selected on the condition that no other alternative operational modes have been determined. For example, the default operational component 10720 may be selected on the condition of a failure of the initialization component and / or an interaction failure. The initialization component 10716 may instruct the operation pointer 10724 to instruct the resident operation component 10722 to operate the upgradeable element 10714. For example, certain characteristics may reside in the upgradeable element 10714, but require startup to operate. The initialization component 10716 may instruct the operation pointer 10724 to instruct the upgradeable element 10714 to install a new operation component 10728 and / or the newly installed operation component 10726. For example, new software and / or firmware may be downloaded. The new software and / or firmware may include code that enables the characteristics represented by the selected operating mode. For example, a new hardware component can be installed to enable the selected operating mode.

[0187] The surgical hub may have collaborative interaction with one of several means of displaying images from surgical scopes, such as laparoscopes, and information from one of several other smart devices. The hub may also be configured to interact with multiple displays to enable combined display and control of data distributed across multiple displays.

[0188] The display of information can be controlled by different visualization control modes. For example, the content on one or more displays can be controlled by the user and / or automated. The visualization control modes can operate at different levels based on the control scheme present in the operating room.

[0189] The display of information from surgical instruments and hubs can operate with multiple levels of complexity and control. These multiple levels can be associated with multiple levels of hardware capacity, software capability, and / or firmware capability. For example, a visualization control mode with more supported capabilities may require interlocking hardware and / or software to ensure timely data synchronization or pairing. These levels can be controlled or limited through different visualization control modes. For example, the current visualization control mode may be determined based on the hub's ability to operate the surgical instrument at an appropriate refresh rate, processing requirements, memory requirements, user input, and / or the purchase level of software subscriptions required to operate the surgical system.

[0190] The hub can adjust the visualization control mode, for example, by upgrading or downgrading, based on the internal parameters of the surgical hub. The internal control parameters may be determined based on changes in these parameters. Changes may be triggered by processing power, available processing power or memory, heat generated by the system, its power consumption, the balance of power consumption with other attached systems, user input, and / or the system's subscription level.

[0191] Figure 37 shows an exemplary flow for a surgical hub operating under a stepped visualization control mode. The hub may include a primary display, a secondary display, a laparoscope, and a communication array that can be connected to at least one surgical instrument. As shown in the figure, in 17501, the hub may acquire one or more visualization control parameters associated with the primary and secondary displays.

[0192] As shown in Figure 37, in 17502, the hub can determine the visualization control mode based on visualization control parameters. The visualization control parameters may include at least one of the following: available memory, available data bandwidth, heat generated by the surgical hub, heat generated by the secondary display, power capacity associated with the surgical hub, power capacity associated with the operating room, power capacity associated with the medical facility, power consumption, balance of power consumption to at least one auxiliary system, processor utilization, or memory utilization.

[0193] In 17503, the hub can generate visualization data for the primary and secondary displays according to the visualization control mode.

[0194] For example, visualization control parameters may include instructions from a layered system. The layered system can scale display capabilities and interactive display control capabilities based on available data bandwidth, power capacity and usage, processor and memory utilization, and / or internal or auxiliary systems. The layered system can determine the maximum display and interactive display control capabilities that the surgical hub can operate under. For example, if the layered system detects that the power capacity associated with the operating room, surgical hub, and / or medical facility falls below a threshold, it may scale down the visualization control capabilities of the surgical hub. For example, if the layered system detects that the available data bandwidth falls below a threshold, memory utilization exceeds a certain threshold, power usage exceeds a certain threshold, and / or other system conditions that may cause a scaling down of visualization control capabilities, it may restrict or disable display-related communications between the surgical hub and the device, and / or between the surgical hub and an external server. Multiple display capabilities may be disabled. Augmented reality capabilities may be disabled. The layered system may be a module within the surgical hub, or it may be a system outside the surgical hub.

[0195] In an exemplary visualization control mode, multiple displays may be used to show different aspects or types of information related to the primary inspector on each display. Some or all of the displays may be controlled by another system with which the hub can communicate.

[0196] In exemplary visualization control modes, one or a portion of one display may be controlled via another display. Content displayed on one or a portion of one display may be associated with the other display. For example, in picture-in-picture display, the content source of a minipicture may be controlled in exemplary visualization control modes. This characteristic is further described in U.S. Patent Application No. 15 / 940,742, filed March 29, 2018, entitled “DUAL COMS ARRAY IMAGING,” which is incorporated herein by reference in its entirety.

[0197] In an exemplary visualization control mode, individual users may have different display systems that function in conjunction with the main shared display. Different overlay information may be generated for different user roles so that users may be provided with personally directed or personalized overlay data. For example, as described in U.S. Patent Application No. 15 / 940,671, filed March 29, 2018, entitled “DUAL SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATION THEATER,” which is incorporated herein by reference in its entirety.

[0198] In an exemplary visualization control mode, the hub can restrict visualization displays to reside on the primary display. For example, when operating under the first visualization control mode, the hub can control the primary display. The hub can determine which information and video displays should share a portion of the entire actual area of ​​the display.

[0199] Figure 16 shows an exemplary primary display 6200 associated with a surgical hub 206, comprising a global display window 6202 and a local instrument display window 6204, according to one aspect of the present disclosure. Referring subsequently to Figures 1 to 11, which show interaction with an interactive surgical system 100 environment including surgical hubs 106, 206, and Figures 12 to 14, which show instruments connected to the surgical hub, the behavior of the local instrument display 6204 may be displayed when the instrument 235 senses that the global display window 6202 is connectably present via the surgical hub 206. The global display window 6202 may show a field of view 6206 of the surgical site 6208, for example, at the center of a surgical hub display 215, also referred to herein as a monitor, as seen through a medical imaging device such as a laparoscope / endoscopy 219 connected to an imaging module 238. The end effector 6218 portion of the connected instrument 235 may be shown within the field of view 6206 of the surgical site 6208 in the global display window 6202. The image displayed on the display 237 located on the instrument 235 connected to the surgical hub 206 is, for example, displayed on or mirrored on the local instrument display window 6204 located in the lower right corner of the monitor 6200, as shown in Figure 16.

[0200] During operation, the relevant instruments, information, and menus may be displayed on the display 237 located on the instrument 235 until the instrument 235 senses its connection to the surgical hub 206, at which point all or a subset of the information presented on the instrument display 237 may be displayed (for example, only on the local instrument display window 6204 portion) of the surgical hub display 6200 via the surgical hub 206. The information displayed on the local instrument display window 6204 may be mirrored on the display 237 located on the instrument 235, or may no longer be accessible on the instrument display 237's display screen. This technique frees the instrument 235 to display different information or to display larger font information on the surgical hub display 6200.

[0201] The primary display 6200 may provide perioperative visualization of the surgical site 6208. High-level imaging may identify and visually highlight important structures such as the ureter 6220 (or nerves, etc.) 6222, tracking instrument proximity indicators 6210 which may be shown to the left of the display 6200. In the illustrated example, instrument proximity indicators 6210 may indicate instrument-specific settings. For example, the upper instrument proximity indicator 6212 may indicate settings for a unipolar instrument, the middle instrument proximity indicator 6214 may indicate settings for a bipolar instrument, and the lower instrument proximity indicator 6212 may indicate settings for an ultrasound instrument.

[0202] Figure 17 shows an exemplary primary display having a composite overhead view of a portion of the end effector 6234 of a surgical stapler mapped using two or more imaging arrays or one array and time, to provide multiple viewpoints of the end effector 6234 and enable composite imaging of the overhead field of view. This technique described herein can be applied to ultrasound instruments, electrosurgical instruments, combinations of ultrasound / electrosurgical instruments, and / or combinations of surgical staplers / electrosurgical instruments. Several techniques can be performed to present a composite image on a display (e.g., a single display) by overlaying or extending images and / or text from multiple image / text sources.

[0203] As shown in Figure 17, the primary display 6200 of the surgical hub 206 can display a primary window 6230. The primary window 6230 may be located in the center of the screen and may show a magnified or resolved narrow-angle view of the surgical field 6232. The primary window 6230 located in the center of the screen shows a magnified or resolved narrow-angle view of the end effector 6234 of a surgical stapler that grasps a blood vessel 6236. The primary window 6230 can display an organized image to produce a composite image that enables visualization of structures adjacent to the surgical field 6232. A second window 6240 may be located in the lower left corner of the primary display 6200. The second window 6240 displays an organized image of a wide-angle view at standard focus of the image shown in the primary window 6230 in an overhead view. The overhead view provided in the second window 6240 allows the examiner to easily view items outside the narrow surgical field of view 6232 without moving the laparoscope or other imaging device 239 connected to the imaging module 238 of the surgical hub 206. The third window 6242 may be shown in the lower right corner of the primary display 6200 and shows an icon 6244 representing the staple cartridge of the end effector 6234 (for example, a staple cartridge in this example), along with additional information such as "4 rows" indicating the number of staple rows 6246 and "35 mm" indicating the distance the knife has traveled along the length of the staple cartridge 6248. Below the third window 6242 is an icon 6258 of the frame of the current state of the clamp stabilization sequence 6250, which indicates clamp stabilization.

[0204] In exemplary visualization control modes, the display may be controlled by the user, for example, via motion tracking (e.g., head orientation relative to the monitor, hand gestures, voice activation, and other means within the sterile field). User gestures may be determined based on wearable devices worn by the user, such as a smartwatch and / or camera within the OR. User head movements may be determined based on AR goggles and / or camera within the OR.

[0205] Figure 19 is a diagram of an exemplary OR setup that may enable display control via motion tracking, gesture tracking, and / or voice activation. In various embodiments, the surgical hub 211801 may be communicably connected to one or more cameras 211802, surgical instruments 211810, displays 211806, overheard lights 211808, and other surgical devices within the OR 211800 via a communication protocol (e.g., Bluetooth). Camera 211802 may be oriented to capture images and / or videos of surgical staff 211803 and / or surgical instruments 211810 (or other surgical devices) within the OR 211800 during the course of a surgical procedure. Captured images may include still images or moving images (e.g., video). Images of surgical staff 211803 and / or surgical instruments 211810 may be captured at various angles and magnifications, utilizing various filters, etc. For example, camera 211802 may be positioned within OR 211800 to allow for the collective visualization of each surgical staff member performing a procedure. Thus, the surgical hub 211801 can receive captured image and / or video data from camera 211802 to visually analyze the surgical staff member 211803 and / or surgical instruments 211810 during the surgical procedure. The image and / or video data can be processed using various machine vision, image processing, object recognition, and optical tracking techniques to track the characteristics, features, actions, and movements of the surgical staff member 211803 and / or surgical instruments 211810.

[0206] Figure 20 is a block diagram of a gesture recognition system 211500 that may be used to control the display in an exemplary visualization control mode. The gesture recognition system 211500 includes a gesture recognition module 211504 that can be executed by a processor or control circuit of a computer system, such as the processor 244 of the surgical hub 206 shown in Figure 10. Thus, the gesture recognition module 211504 can be embodied as a set of computer-executable instructions stored in memory 249, which, when executed by the processor 244, cause a computer system (e.g., the surgical hub 211801) to perform the steps described.

[0207] The gesture recognition system 211500 receives image or video data from image recognition hardware / software (e.g., camera 211802), recognizes various gestures 211804 that may be performed by surgical staff 211803 (e.g., processes 211600, 211620 determine whether a gesture is being performed (211604, 211624)), and can perform a corresponding action or respond in another way to a specific detected gesture 211804 (e.g., processes 211600, 211620 control surgical equipment (211606), or store the data as metadata (211626)). In one embodiment, the gesture recognition module 211504 may include a feature extraction module 211506 and a gesture classification module 211508. The feature extraction module 211506 can extract measurable and discriminant traits or features (e.g., characteristics) from image / video data. Features may include edges (extracted via, for example, the Canny edge detection algorithm), curvature, corners (extracted via, for example, the Harris & Stephens corner detection algorithm), etc. The gesture classification module 211508 can determine whether the extracted characteristics match gestures from a set of gestures. In one embodiment, the gesture classification module 211508 may include a machine learning model (e.g., an artificial neural network or support vector machine) trained via supervised or unsupervised learning techniques to correlate the characteristic vectors of the extracted characteristics to one or more output gestures. In another embodiment, the gesture classification module 211508 may include an algorithm based on Hu invariant moments or a k-curvature algorithm to classify gestures. In yet another embodiment, the gesture classification module 211508 may include a template matching algorithm programmed to match characterized image / video data (or a portion thereof) to a template corresponding to a given gesture. Other embodiments may include various combinations of the aforementioned techniques with other techniques for classifying gestures.

[0208] When a gesture is recognized via the gesture recognition module 211504, the gesture recognition system 211500 can perform an action 211510 or response corresponding to the identified gesture. For example, an action 211510 performed by a computer system may include controlling a surgical display within the OR.

[0209] Action 211510 performed by the computer system may include storing gestures made by surgical staff as metadata associated with or linked to perioperative data generated by surgical instruments during the course of the surgical procedure. Such metadata may be useful in determining whether surgical staff are controlling surgical instruments manually or via gestures, and the gestures can be correlated with surgical staff performance, procedure time, and other such metrics. In various other embodiments, the computer system may both control one or more surgical instruments and store gesture data as metadata.

[0210] The gesture recognition system 211500 may, in addition to or instead of the camera 211802, utilize a magnetic sensing system to receive non-contact input from the user in order to visually identify gestures. In this embodiment, the gesture recognition system 211500 may include, for example, a magnetic sensing array that can be positioned within the OR.

[0211] Gesture recognition is further described in U.S. Patent Application No. 16 / 182,269, filed November 6, 2018, entitled "IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE" (Agent Reference Number: END9018USNP3), which is incorporated herein by reference in its entirety.

[0212] Figure 38 shows a detailed exemplary flow of hub operation under a hierarchical visualization control mode. The hub can acquire the visualization control mode in 17510. In 17511, the hub can generate and transmit data to the primary display as described herein. In 17512, the hub can decide whether to generate visualization data for the secondary display based on the visualization control mode.

[0213] Some exemplary visualization control modes can support multi-display capability, while other exemplary visualization control modes can restrict visualization to being on the primary display, or to display the same content on both the primary and secondary displays. If a visualization control mode supports multi-display capability, in 17513, the hub can generate visualization data for the secondary displays and transmit the generated visualization data to each secondary display. If a visualization control mode does not support multi-display capability, in 17514, the hub can disable the generation and transmission of visualization data for the secondary displays and continue to transmit data to the primary display.

[0214] Figure 40 shows an exemplary flow for a hub operating under a visualization control mode that supports multi-display capability. In 17601, the hub can acquire display control parameters associated with a surgical procedure. Display control parameters may include at least one of the following: user orientation to at least one display, progress of the surgical procedure, surgical context, and / or detection of anomalies associated with the surgical procedure. For example, display control parameters may be voice commands, user input via an interactive display, content type, intended examiner of the displayed information, and / or content of the information to be displayed.

[0215] In 17602, the hub can determine different content for different displays based on display control parameters. In 17603, the hub can generate display content and send it to each display.

[0216] For example, the display control parameter may be the user's orientation to the display. The surgical hub may determine the display content and / or display format on one or more displays based on the orientation of the primary surgeon's head (or the user for whom the information is useful) to the displays in the OR. The surgical hub may also determine the display content and / or display format on one or more displays based on user input, including user input from inside or outside the OR. For example, the surgical hub may determine the display position, such as identifying a display, or identify a display window within a display, based on the intended examiner of the information and the examiner's relative position to one or more displays in the OR (e.g., each display). For example, the surgical hub may select the display closest to the intended examiner of the information. The surgical hub may decide to remove certain display content based on the intended examiner of the information and the viewer's relative position to the various displays in the OR.

[0217] In various embodiments, controls for surgical hubs, surgical instruments, and other devices may be adjusted based on a screen operating on a sterile field display. Controls for surgical devices may be adjusted based on the displayed information. Controls that normally control the panning of a visualization device (e.g., a scope) or the focusing of a visualization device may be configured, for example, to adjust the magnification when a hyperspectral imaging overlay is active. Hyperspectral imaging is further described in U.S. Patent Application No. 15 / 940,722, filed March 29, 2018, entitled "Characterization of Tissue Irregularities Through the Use of Mono-Chromatic Light Refractivity," which is incorporated herein by reference in its entirety.

[0218] For example, a surgeon can control the data on the display to modify, focus, or manipulate it. This could allow medical professionals to see more seamlessly where they are relative to other imaging or pre-operative imaging mechanisms.

[0219] Controls on the handles of surgical instruments within the field of view of the sterile field display can be adjusted by selection on the sterile field display. Furthermore, adjustments can be based on situational awareness in various cases. For example, the system may determine that a particular surgical device is being used and that the system is permitted to control the functions of that surgical device from a second device, such as a display screen within the sterile field.

[0220] In an exemplary visualization control mode that supports collaborative display capabilities, multiple displays can be used to display different aspects or types of information relevant to the primary inspector of each display. Some or all of the displays may be controlled by another system that only communicates with the main hub, rather than being controlled by it.

[0221] Multiple displays may include, but are not limited to, a primary display on a hub, a visualization tower which may include at least one monitor, displays around the room, and / or small device displays.

[0222] In an exemplary visualization control mode supporting collaborative display capabilities, the surgical hub may enable medical professionals to control displays outside the sterile field via displays inside the sterile field. During surgical procedures, surgeons may not have access to user interface devices for interactive input and displays within the sterile field. Therefore, surgeons may not interface with user interface devices and the surgical hub from within the sterile field, and may not be able to control other surgical devices through the surgical hub from within the sterile field.

[0223] For example, a local display, such as a secondary display, may serve as a user interface for displaying and controlling surgical hub functions from within the sterile field. The secondary display can be used to change the display position, what information is displayed where, and to pass control of a specific function or device. The local display may include a display unit used within the sterile field, which may have interactive input control from the sterile field to control other surgical devices and / or displays connected to the surgical hub. The display unit may be sterile and may be located within the sterile field so that the surgeon can interface with the display unit and the surgical hub and, if necessary, interface directly with instruments and configure them without leaving the sterile field. The display unit may also be a master device and may be used for display, control, and tool control exchange, enabling the surgeon to feed from other surgical hubs without leaving the sterile field.

[0224] The display unit may be an interactive touchscreen display, an interface configured to connect the interactive touchscreen display to a surgical hub, a processor, and memory connected to the processor, or may include these. The memory can store instructions that can be executed by the processor to receive input commands from the interactive touchscreen display located inside the sterile field, and can send input commands to the surgical hub to control the device connected to the surgical hub located outside the sterile field.

[0225] The display outside the sterile field may be, or include, a non-sterile display 107 or 109 as shown in Figure 2. For example, the display inside the surgical sterile field may be, or include, a secondary display such as a local display or a display on a surgical instrument. A healthcare professional can control the secondary display. The primary and secondary displays may have multiple levels of communication with the primary hub system. Examples of primary and secondary displays can be found in more detail in U.S. Patent Application No. 15 / 940,671, “SURGICAL HUB SPATIAL AWARENESS TO DETERMINES DEVICES IN OPERATING THEATER,” filed March 29, 2018 (Agent Reference No. END8502USNP), which is incorporated herein by reference in its entirety.

[0226] An example of controlling an external display via an internal display within a sterile field is described in the concurrently filed patent application, attorney reference number END9287US17, entitled "COMMUNICATION CONTROL OPTIONS FOR A SURGEON CONTROLLED SECONDARY DISPLAY AND PRIMARY DISPLAY," which is incorporated herein by reference in its entirety.

[0227] The secondary display may include a touchscreen display capable of displaying and / or providing status for any number of surgical hub 206 tracking data feeds, and / or a separate secondary display and / or a dedicated local display that can be linked to the surgical hub 206 to provide an interaction portal via the secondary screen. The secondary display may display force to fire (FTF), interstitial space, power level, impedance, tissue compressive stability (creep), etc., while the primary display may display variables important for keeping the feed clutter-free. The interactive display may be used to move the display of specific information to the primary display in a desired position, size, color, etc. In the illustrated example, the secondary display may display an instrument proximity indicator 6210 to the left of display 6200. The local instrument indicator 6204 is located on the lower right side of display 6200. The local instrument display 6204 presented on the surgical hub display 6200 may display icons for the end effector 6218, such as an icon for the staple cartridge 6224 currently in use, the size 6226 of the staple cartridge 6224 (e.g., 60 mm), and an icon for the current position of the end effector knife 6228.

[0228] The secondary display may be a display 237 as shown in Figures 5 and 6. Referring to Figure 6, the display 237 located on the instrument 235 can display the wireless or wired mounting of the instrument 235 to the surgical hub 206, as well as the instrument's communications and / or records on the surgical hub 206. Settings may be provided on the instrument 235 to allow the user to choose to mirror or extend the displays on both monitoring devices. Instrument controls may be used to interact with the surgical hub display of the information supplied on the instrument. The instrument 235 may include a wireless communication circuit for wireless communication with the surgical hub 206, as described herein.

[0229] A first instrument connected to the surgical hub 206 can be paired with the screen of a second instrument connected to the surgical hub 206, allowing both instruments to display some hybrid combination of information from both devices, which mirror portions of the primary display. The primary display 6200 of the surgical hub 206 can provide a 360° composite top view of the surgical site 6208 to avoid collateral structures. For example, a secondary display of an end-effector surgical stapler may be provided within the primary display 6200 of the surgical hub 206, or on a separate display, to provide better perspective around areas within the current field of view 6206.

[0230] This secondary display can also be used as a control mechanism to adjust what information is displayed and how it is displayed on the primary display outside the sterile field. This would allow for better highlighting of other surgical personnel information that needs to be tracked, recognized, or assisted.

[0231] These secondary displays may be located on the instrument, positioned above the patient adjacent to the surgical access port, or attached to the user. These displays can modify multispectral imaging, control its overlay on the normal scope feed, overlay preoperative imaging based on established positional characteristics, adjust axillary data displayed around the display, or its order or size, and even allow the user to move one image or dataset from one position to another on another display.

[0232] The primary and secondary displays may be controlled via a gesture recognition system such as those described herein.

[0233] For example, the visualization control parameter may be the progress of a surgical procedure. The surgical hub may determine the display content for the primary and secondary displays based on the progress of the surgical procedure.

[0234] Visualization control can be adjusted according to the steps of the surgical procedure being performed. Situational awareness can notify the surgical hub of the current and / or next steps of the surgical procedure. For example, based on previous surgical actions and / or usage sequences of surgical devices and / or generators, the surgical hub can determine which specific step of a particular surgical procedure is being performed, such as whether the procedure is currently in the nodule incision step, the transverse vascular section step, or something else. The surgical hub and / or generators can determine procedure-specific steps or contexts.

[0235] For example, surgical context data may include the type of surgical procedure being performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This ability, relating to some aspects of a surgical hub for deriving or inferring information about a surgical procedure from received data, is sometimes referred to as “context awareness.” In one example, a surgical hub may incorporate a context awareness system, as described herein with reference to Figures 9 and 10. A context-aware surgical hub can derive context information about a surgical procedure from various received surgical data. Such surgical data may include perioperative data from modular devices 5102 and other data sources (e.g., database 5122 and patient monitoring device 5124) that are communicably connected to the surgical hub 5706.

[0236] As described herein, the hub can learn and predict procedure-specific steps or contexts by analyzing the most common usage of a particular clinician at each stage of a surgical procedure and / or after the exchange of a certain number or type of surgical instruments. After monitoring the behavior of the same clinician over a predetermined number of procedures including the same steps, the hub may automatically change the content displayed on the display based on the monitored past displays, the clinician's interactions with the displays, and / or the controls indicated by the clinician. In various examples, the hub can provide notifications to the clinician when the displays are adjusted. For example, the hub and / or displays can provide auditory notifications (e.g., beeps or verbal explanations), visual cues (e.g., flashing lights and / or words on the screen), and / or tactile warnings (e.g., vibration and / or movement of surgical instruments, or parts of surgical instruments such as the actuator buttons themselves). In other examples, the surgical hub can recommend display adjustments. Recommendations from the surgical hub are described further herein.

[0237] Figure 41 shows an exemplary flow for a hub operating under a visualization control mode that supports situational awareness. At 17610, the hub can acquire a visualization control mode associated with a surgical procedure. At 17611, the hub can receive perioperative data from at least one surgical instrument. At 17612, the hub can determine surgical progression based on the visualization control mode and at least a portion of the perioperative data.

[0238] The progress of a surgical procedure may be determined using a situation-aware surgical system 5100, as shown in Figures 9 and 10. For example, the situation-aware hub 5104 can determine which steps of a surgical procedure are being performed or will be performed. Based on the received data, the situation-aware hub 5104 can determine whether an event has occurred. An event may include, for example, a surgical procedure, a step or part of a surgical procedure, or a pause in a surgical procedure or between steps of a surgical procedure. Furthermore, the surgical hub 5104 can track data associated with a particular event, such as the duration of the event, surgical instruments and / or other medical products used during the course of the event, and the medical personnel associated with the event. The surgical hub 5104 may determine event data through a situation-aware process, for example, as described herein. The situational awareness process is described in U.S. Patent Application No. 15 / 940,654, titled "SURGICAL HUB SITUATIONAL AWARENESS," filed on March 29, 2018 (Agent Reference Number END8501USNP), U.S. Patent Application No. 16 / 209,478, titled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE," filed on December 4, 2018 (Agent Reference Number END9015USNP1), and "ADJUSTMENTS BASED ON AIRBORNE PARTICLE More details are provided in U.S. Patent Application No. 16 / 182,246, entitled “PROPERTIES” (Agent Reference No. END9016USNP1), the entirety of which is incorporated herein by reference.

[0239] Referring back to FIG. 41, based on the determined surgical procedure and the type of display, the hub may then determine the display content at 17613. At 17614, the hub can instruct the display to display the determined display content.

[0240] For example, the hub can associate different display contents with different exemplary treatment steps shown in FIG. 22. As shown, the exemplary surgical steps can include separating the lungs, managing the major blood vessels, and removing the lobes. The surgical hub may instruct the display to show information particularly relevant to the current step in the surgical procedure based on situation recognition and automatic control. The surgical hub can determine the type of surgical data for display based on the determined progress of the surgical procedure. The surgical hub may select a display from within the OR displays to display the surgical database during the determined progress of the surgical procedure.

[0241] For example, baseline visualization of the anatomical structure and / or the surgical site can be obtained prior to the start of the surgical procedure, such as before tissue manipulation and incision at the surgical site. The baseline visualization image of the anatomical geometry can include visualization of the surface and its boundaries of the anatomical structure. Such a baseline visualization image can be used to maintain the overall orientation of the surgical site and the anatomical structure even when local regions within the anatomical structure are gradually disrupted, altered, or otherwise manipulated during the surgical procedure.

[0242] For example, a surgical hub can update baseline visualization images by identifying a specific type of surgical procedure, surgical procedure steps, tissue type, and / or one or more specific tissue features. In one example, the updated baseline visualization image may be useful after a transverse incision or after the application of one or more staple rows. In a particular example, distorted subregions within the original anatomical structure may generate a separate new baseline visualization image, or the existing baseline visualization image of the distorted subregion may be updated to appropriately indicate image overlays. For example, important areas of a patient's anatomical structure may be updated after tumor removal or growth within it.

[0243] For example, a surgical hub may use spectral imaging techniques to generate display content to visualize different tissue types and / or anatomical structures, as shown in Figure 23. In Figure 23, a spectral emitter 2320 (e.g., spectral light source 150) may be used by the imaging system to visualize the surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissues and / or structures of the surgical site 2325 is received by an image sensor, which can visualize the tissues and / or structures, which may be either visible (e.g., located on the surface of the surgical site 2325) or occluded (e.g., beneath other tissues and / or structures in the surgical site 2325). In this example, the imaging system can visualize tumors 2332, arteries 2334, and various anomalies 2338 (i.e., tissues that cannot be identified against known or assumed spectral signatures) for each of the various tissue / structure types, based on spectral signatures characterized by different absorption characteristics (e.g., absorption coefficients) of the constituent materials. The visualized tissues and structures can be displayed on display screens associated with or linked to the imaging system, such as the imaging system display, primary display, non-sterile display, hub display, and device / instrument display.

[0244] The surgical hub can adjust or update the visualization of the displayed surgical site according to the identified tissue and / or structure type. For example, a margin 2330a associated with the visualized tumor 2332 can be displayed on the screen. The margin 2330a can indicate the area or amount of tissue to be excised to ensure complete removal of the tumor 2332. The control system can be configured to control or update the dimensions of the margin 2330a based on the tissue and / or structure identified by the imaging system. In the illustrated example, multiple anomalies 2338 can be identified within the FOV. Therefore, the control system can adjust the displayed margin 2330a to match a first updated margin 2330b that has sufficient dimensions to encompass these anomalies 2338. Additionally, arteries 2334 can be identified that partially overlap with the initially displayed margin 2330a (shown in the highlighted area 2336 of artery 2334). The surgical hub can adjust the displayed margin 2330a to match a second updated margin 2330c that has sufficient dimensions to encompass the relevant portion of artery 2334.

[0245] For example, if it is determined that the next surgical step is to excise a portion of tissue, the surgical hub may display on the display an estimated change in deformation for the proposed excision. The proposed excision line may be added to the digital model, which may be updated to show the anatomical structure with the virtual excision. Referring again to Figure 13B, in one example, the clinician may intend to remove a wedge-shaped portion from the tissue of the surgical site 2325 in order to remove the tumor 2332 along with the tissue anomaly 2338. In such a case, the model may be updated to show the organ from which the wedge-shaped portion has been removed. The updated model may depict the deformation of the tissue, as well as the calculated stress and / or strain within the tissue based on the known mechanical properties of the tissue and the deformation induced by the surgery. For example, the tissue may be shaded or otherwise layered with stress and / or strain data so that the clinician is informed about how a particular excision may affect the strain on the tissue. In some embodiments, stress / strain data may be overlaid on the image as a set of vector lines indicating the direction and line type or color of the stress / strain, in order to show the stress / strain values. Based on the calculated stress and strain, the clinician may modify the proposed excision and consider alternative strategies to reduce and / or better distribute the stress and strain within the tissue. For example, the angle of the excision may be modified. In certain cases, the clinician may reorient the staple lines in a preferred strain direction.

[0246] For example, if it is determined that the surgical procedure is a video-assisted thoracoscopic (VATS) procedure, the surgical hub can instruct one or more displays to show exemplary content as shown in Figures 24 and 25. A VATS procedure is a surgical procedure in which one or more surgical instruments and one or more thoracoscopes (i.e., cameras) are inserted into the patient's thoracic cavity through a slit positioned between the patient's ribs. The cameras are used to provide the surgeon with a view of the inside of the patient's thoracic cavity, enabling the surgeon to properly position / move the surgical instruments and manipulate the tissues / structures within the thoracic cavity. Because the surgeon controls the surgical instruments based on what is displayed by the imaging system via the camera, and because the surgical instruments may not be aligned with the camera's viewpoint, the spatial relationship between the surgical instruments and the POV displayed by the imaging system can potentially be misoriented, especially in the case of imaging systems that allow the user to pan, manipulate, and reorient the displayed visualization.

[0247] Figures 24 and 25 show exemplary display content associated with a VATS procedure. In this particular VATS procedure, the surgeon may attempt to remove a tumor 6506 located within the apical segment of the upper lobe of the lung 6508. As illustrated, the surgeon positions a port 6502 between a second rib 6501 and a third rib 6503 to provide an access route 6504 for a surgical instrument 6510 (e.g., a surgical stapler) insertable through port 6502 to access the tumor 6506 and / or surrounding areas within the pleural cavity. Once the access location for the surgical instrument 6510 is selected, the surgeon may position one or more cameras 6520a, 6520b through other ports 6502 positioned to allow the cameras 6520a, 6520b to visualize the interior of the pleural cavity near the surgical site. Visualizing the surgical site in this manner allows the surgeon to position and orient the end effector 6514 of the surgical instrument 6510 and manipulate the tissue as needed (for example, to excise a portion of the lung 6508 around a tumor 6506). In the particular illustrated example, two cameras 6520a and 6520b are used, but a different number of cameras may be used, and / or one or more of the cameras 6520a and 6520b may be oriented in different ways depending on the particular type of surgical procedure being performed and / or the area of ​​the patient 6500 body that needs to be visualized.

[0248] For example, when operating under exemplary visualization control modes, the surgical hub can adjust a secondary display, such as a local display attached to a surgical instrument, based on a local coordinate system. The local coordinate system may be a surgical visualization coordinate system. When it is determined that the surgical procedure is a VATS procedure, the surgical hub may transmit the locally displayed coordinate system to the surgical instrument or other medical device so that the instrument / device controls are adapted to control movement relative to the local visualization coordinate system. At least one measurement derived from the imaging system may be used to define the local coordinate system. User controls displayed on the local display may be reoriented relative to the local coordinate system rather than to a standard global coordinate system or another coordinate system.

[0249] As shown in Figure 25 and listed below in Table 1, various different coordinate systems can be defined with respect to different points of view (POV) of the patient, device, or device components. Furthermore, when operating under a visualization control mode that allows the user to manipulate the displayed visualization, a “virtual” POV corresponding to the virtual or predictive visualization displayed to the surgeon can be defined, and the coordinate system can also be defined according to these POVs. The generation and control of such visualizations will be further described herein.

[0250] [Table 1]

[0251] A coordinate system may be defined based on sensor measurements and / or measurements by an imaging system. For example, the coordinate system for a surgical instrument handle assembly 6512, a shaft 6513, or an end effector 6514 may be defined according to measurements by an accelerometer or another such sensor associated with each component. As another example, any of the aforementioned coordinate systems may be defined based on measurements of the relative distance and / or position of objects relative to one another, or they may be a global coordinate system determined by imaging the objects via an imaging system.

[0252] In the example shown in Figure 26, the surgical instrument 6510 determines, using the provided transformation function, that the control 6518 and the display screen 6516 should be adjusted based on the updated coordinates. In various examples, situational awareness can be notified when the control 6518 and / or the display screen 6516 are updated, as further described herein. The display screen 6516 can display the GUI 6517 adjusted from a first orientation shown on the left side of Figure 26 to a second orientation shown on the right side of Figure 26, so that the GUI 6517 is properly oriented for the surgeon to control the surgical instrument 6510. In one embodiment, the GUI 6517 may further include a GUI element 6524 (e.g., an icon) indicating the POV or coordinate system used by the surgical instrument 6510. In this example, GUI element 6524 shifts to indicate that the POV displayed by the visualization system 2108 has changed from the device coordinate system ("DVC") to the local coordinate system ("local") associated with the image / video displayed by the visualization system 2108.

[0253] As an example, the surgical instrument control 6518, adjusted according to the updated coordinates, may include joint movement controls. The joint movement controls may include, for example, a first control 6519a configured to articulate the surgical instrument 6510 in a first direction, and a second control 6519b configured to articulate the surgical instrument 6510 in a second direction. The joint movement controls 6519a, 6519b may be embodied, for example, as rockers, toggles, or separate actuators and / or buttons. In this example, the surgical instrument 6510 causes the first joint movement control 6519a and the second joint movement control 6519b to swap functions in response to a change in the orientation of the surgical instrument 6510. In other words, activating the first joint movement control 6519a will instead cause the surgical instrument 6510 to move jointly in a second direction, and activating the second joint movement control 6519b will cause the surgical instrument 6510 to move jointly in a first direction. Therefore, the functions of the joint movement controls 6519a and 6519b can be set according to the orientation of the surgical instrument 6510 or its components (e.g., end effector 6514) displayed to the user.

[0254] In addition or alternatively, in certain examples, a GUI 6517 on the display screen 6516 may be adjusted. For example, the GUI 6517 may be inverted when the handle assembly 6512 is inverted. In certain examples, the GUI 6517 may include a touchscreen so that the surgeon can switch between coordinate systems by interacting with the GUI 6517. For example, the surgeon can switch between the device POV, the local POV, and / or one or more other POVs by interacting with the GUI 6517.

[0255] When operating under exemplary visualization control modes, the surgical hub may, for example, determine that the current surgical procedure could benefit from an expanded visualization field of view, and may therefore fuse images from different sources to expand the visualization field of view. For example, if the surgical hub determines that the current surgical procedure is to incise a blood vessel, it may generate and transmit a fused image from different sources.

[0256] A 3D representation of an object within the visualization field of view of an imaging system may be created, and the 3D shape may be characterized to allow the user to modify the displayed visualization relative to an established coordinate system in order to better visualize the surgical site. The 3D representation may be generated from images produced from a real-time source or a non-real-time source (e.g., a CT scan or MRI). In one embodiment, structured light or structured EMR can be projected to create a structured 3D shape that can be tracked in real time. These 3D shapes may be generated so that the POV displayed by the display is moved or rotated away from the local coordinate system of the scan source in order to improve the user's viewpoint through the display.

[0257] Figure 27 shows an exemplary FOV 6570 of the camera during a VATS procedure. The target of this particular exemplary procedure is a tumor 6506 located within the apical segment of the upper lobe 6580 of the lung 6508. Many biological structures are identifiable within this FOV 6570, including the chest wall 6509, veins 6574, arteries 6576, bronchi 6578, fissures 6582 depicting the upper lobe 6580, pulmonary artery 6584, and pulmonary veins 6586. Non-biological objects are also visible within FOV 6570, including the end effector 6514 and shaft 6513 of a surgical instrument 6510 controlled by the surgeon. In the exemplary imaging system, such a view would be the only view available to the surgeon performing the video-assisted procedure, combined with any corresponding views from any additional cameras 6520 being used. While these cameras are positioned to provide surgeons with an appropriate field of view for performing surgical procedures, the field of view provided by camera 6520 may ultimately not provide an ideal FOV 6570 for performing each step or task in the surgical procedure, or unexpected obstructions may be present at the surgical site that obstruct the surgeon's view. Furthermore, repositioning or reorienting camera 6520 during surgery may be impractical or undesirable in some cases due to surgical constraints of the procedure.

[0258] The surgical system may be configured to extend the field of view provided by the camera by combining multiple images of the surgical site, including pre-operative and intra-operative images, to generate a 3D representation of the surgical site or the tissues and / or structures located within the surgical site. During the surgical procedure, the user can then manipulate the 3D representation displayed by the imaging system 142 to visualize the surgical site from orientations outside the range of the camera's FOV 6570 used in the procedure. Such a reoriented view may be referred to as a "virtual POV" as described above. Thus, the surgical system complements the FOV 6570 provided by the camera, allowing the surgeon to dynamically adjust the displayed visualization of the surgical site during the surgical procedure to find the ideal viewing POV for performing one or more surgical tasks.

[0259] The coordinate system displayed locally is further described in U.S. Patent Application No. 16 / 729,747, entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS," filed on December 31, 2019 (Attorney Docket No.: END9217USNP1), the entire disclosure of which is incorporated herein by reference.

[0260] FIG. 42 shows an exemplary flow of the operation of the hub in a visualization control mode that supports adjusting a display based on an adjusted display event. At 17620, the hub can receive data from at least one surgical instrument. At 17621, the hub can detect a surgical context based at least in part on perioperative data. The hub can determine, at 17622, whether the surgical context corresponds to an adjusted display event based on the surgical context. If the surgical context includes an adjusted display event, the hub may adjust the display content for one or more displays based on the adjusted display event, as described at 17623. The adjusted display event may include a stressful treatment step, a critical treatment step, and / or a predetermined treatment step.

[0261] For example, if the surgical hub determines that the current surgical step is a stressful treatment step, an important treatment step, or a predetermined treatment step, it may adjust the display format and / or display content on the display to a focus mode.

[0262] Figure 33 shows exemplary procedure steps and progressions that may be detected by the system's exemplary situational awareness capabilities. Certain steps may be considered critical to the success of the surgery or may be associated with a high level of stress. For example, ligating the IMA branches, accessing the plane between the retina and the colon, managing a severely hemophilic patient, and freeing the splenic flexure from the retina, spleen and colon, as shown under the segment “Separate the Colon,” may be considered stressful procedure steps by the surgical hub. As shown in Figure 33, steps such as transversely incising the distal sigmoid colon under the rectosigmoid junction under the segment “Resect the Sigmoid Colon,” and firing the circular stapler under the segment “Pre-form the Anastomosis,” may be considered stressful procedure steps that may cause changes in display content and / or display format.

[0263] For example, the displayed content may be adjusted by zooming in on a target within the image, removing irrelevant information from the first displayed content, and / or highlighting a portion of the laparoscopic scope image.

[0264] Adjusted display events may include the detection of anomalies associated with surgical procedures, the receipt of surgical data being outside the expected range of values, or system parameters being outside the desired system parameter range. Display content may be adjusted by projecting warnings, error messages, or instructions for detected anomalies onto a hub display (e.g., the main monitor). Display content may also be adjusted by overlaying warnings, error messages, or instructions for detected anomalies onto the display.

[0265] Adjusted display events may include detection of a procedure for use being out of sequence. For example, the procedure for using a surgical instrument may be displayed on a device screen, such as a display attached to the surgical instrument. Based on the surgical context, which is at least partially based on the received perioperative data, the situation awareness hub may detect that a procedure for using a surgical instrument is out of sequence. If detected, the display content on the primary display (e.g., the main screen) may be adjusted to indicate instructions for the procedure for using the surgical instrument that is out of sequence. If an early action is identified, the surgical hub may instruct the primary display to show instructions for the recommended procedure. For example, if it senses that the firing trigger has been pulled before the clamp time, the surgical hub may adjust the display content on the primary display to indicate instructions to the user to wait or count down before firing.

[0266] In the example, the displayed content may be adjusted by moving specific data to a different display. The interactive display may receive user instructions, for example, from a medical professional such as a surgeon, indicating a selection of where the data should be displayed. The selection may be indicated for specific surgical procedures, stressful procedures, critical procedures, and / or events in which an anomaly associated with a surgical procedure is detected. The content may be transmitted to the display location selected for display.

[0267] Referring back to Figure 42, if the surgical context does not include a modified display event, the hub may refrain from making additional adjustments to the display in 17624.

[0268] In the example, a hub communicating with an AR device and at least one smart surgical device can provide an interactive overlay for a surgical display that superimposes information onto another surgical display. The surgical display may be connected to the AR device in the operating room. The AR device may overlay or superimpose additional datasets or data streams received from the hub onto a display, such as the surgical display or a display on the smart device. This interactive overlay may allow the user to layer data on the screen as they view the screen. The surgical hub may adjust the layered data based on the display the user is viewing. For example, the hub may adjust the layered data when the user views one display from another. The AR device can adjust the display data on the monitor or device screen. For example, display control instructions may be received from the AR device. In response, the surgical hub may adjust the content to be displayed on the monitor or device screen based on the received display control instructions.

[0269] The AR device may, for example, provide an auditory overlay in addition to listening to the OR sound, rather than replacing the OR sound itself. The AR system can communicate specific information only to targeted individuals within the OR who can access that information.

[0270] AR content can be enabled or disabled based on the location of the AR device. For example, a surgical hub may detect that the AR device is outside the boundaries of the surgical room. In response, the surgical hub may disable the transmission of AR content to the AR device.

[0271] Figure 39 shows an exemplary flow for hub operation under visualization control mode, where the secondary display is an augmented reality (AR) device. At 17520, the hub can acquire visualization control mode. At 17521, the hub can identify the secondary display, which is an AR device.

[0272] As shown in Figure 39, the hub can, at 17522, determine whether to generate overlay information associated with the primary display for overlaying via the secondary display, based on the visualization control mode. If the visualization control mode supports AR capabilities, the hub can, at 17524, disable the generation of information associated with the primary display for overlaying via the secondary display.

[0273] The secondary display may be an AR device, or may include one. The AR device may include a head-mounted display (HMD). The HMD may include a processor, a non-temporary computer-readable memory storage medium, and executable instructions contained within the storage medium that are executable by the processor performing the method or part of the method disclosed herein. The HMD may include a graphics processor for rendering 2D or 3D video and / or images for display.

[0274] Figure 18 depicts a perspective view of a surgeon using a surgical instrument, including a handle assembly housing and a wireless circuit board, during a surgical procedure, with the surgeon wearing a set of safety glasses. The safety glasses may be or may include an augmented reality (AR) device capable of functioning as a secondary display. The wireless circuit board transmits signals to the safety glasses worn by the surgeon using the surgical instrument during the procedure. The signals are received by a wireless port on the safety glasses. One or more indicators on the front lenses of the safety glasses may change color, dim, or incandescent in response to the received signals to indicate information about the status of the surgical instrument to the surgeon. The indicators may be positioned on the periphery of the front lenses so as not to obstruct the surgeon's line of sight. Further examples are described in U.S. Patent No. 9,011,427, “SURGICAL INSTRUMENT WITH SAFETY GLASSES,” filed April 21, 2015, which is incorporated herein by reference in its entirety.

[0275] Figure 18 shows a variation of safety glasses 6991 that a surgeon 6992 may wear while using medical devices during a surgical procedure. When in use, a wireless communication board housed within the surgical instrument 6993 can communicate with a wireless port 6994 on the safety glasses 6991. The exemplary surgical instrument 6993 is a battery-powered device, but the instrument 6993 may be powered by a cable or other means. The instrument 6993 includes an end effector. In particular, the wireless communication board 6995 transmits one or more wireless signals, indicated by arrows (B, C), to the wireless port 6994 of the safety glasses 6991. The safety glasses 6991 receive the signals, analyze the received signals, and display the indicated status information received by the signals on the lenses 6996 to the user, such as the surgeon 6992 wearing the safety glasses 6991. Additionally or alternatively, the wireless communication board 6995 transmits a wireless signal to the surgical monitor 6997 as described above, thereby allowing the surgical monitor 6997 to display the received, indicated status information to the surgeon 6992.

[0276] A variation of the safety glasses 6991 may include a lighting device on the periphery of the safety glasses 6991. The lighting device provides peripheral visual sensory feedback of the device 6993 that the safety glasses 6991 communicate to the user wearing the safety glasses 6991. The lighting device may be, for example, a light-emitting diode ("light-emitted diode, LED"), a series of LEDs, or any other suitable lighting device known to those skilled in the art and apparent to those skilled in the art in consideration of the teachings herein.

[0277] As shown in Figure 39, if the visualization control mode supports AR capabilities, in 17523, when the hub detects that the user of the secondary display is looking at the primary display, it may overlay the overlay information onto the primary display via the secondary display.

[0278] In one example, the primary display may show a live stream of the surgical site in the operating room from a medical imaging device, and the secondary display may be AR glasses. For example, a physician performing laparoscopic surgery while wearing AR glasses can see an image of a tumor overlay on the screen. If the hub detects that the physician is looking down at the patient (e.g., via gesture recognition as described herein, via HMD-based motion tracking, or via image recognition based on images captured by the AR glasses), the hub may instruct the AR glasses to overlay the laparoscopic image with AR content, along with the orientation of the instruments in the patient. This may allow the physician to see the overlay with the orientation of the instruments in the patient. Since the tumor is in three-dimensional space, the physician can only see the outer draping of the tissue with the help of the AR glasses, but the physician can better orient surgical instruments.

[0279] A surgical hub communicating with a specific AR device can generate and transmit different overlays based on the targeted display within the OR. Users can observe different overlays when viewing different displays without interference from each other. The hub can adjust the information contained in the overlays based on different displays in the OR room, specific circumstances, information received from the surgical device, specific user requirements, and / or specific surgical procedures.

[0280] In an exemplary visualization control mode supporting targeted AR content, individual users may have different display devices that can function in conjunction with a shared display. Different display devices may be provided with different AR content to be overlaid on the shared display. This allows users to view personally directed information or overlaid data that only they can see and / or interact with. An exemplary interactive surgical system is described in detail in U.S. Patent Application No. 15 / 940,671, “SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER,” filed March 29, 2018 (Agent Reference Number END8502USNP), which is incorporated herein by reference in its entirety.

[0281] The enhancement of user perception may be, for example, vision via AR glasses or a local display. For example, Figures 28, 34A–34C provide exemplary visual enhancements of user perception. Further examples of visual enhancements are described in U.S. Patent Application No. 15 / 940,704, filed March 29, 2018, entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT" (Agent Reference Number END8504USNP), which is incorporated herein by reference in its entirety.

[0282] The enhancement of the user's perception may, for example, be audible. The audible overlay may be provided via an earphone set with pass-through noise capability and / or via a bone conduction speaker system.

[0283] The surgical hub may adjust visual, audible, and / or other types of user perceptual enhancements based on its situational awareness capabilities as described herein. AR content may be adjusted, for example, based on detected surgical progress. AR content may also be adjusted based on user actions, voice commands, hand gestures, and / or in a predetermined manner. AR devices may be commanded by the user to operate in a pre-customizable manner.

[0284] AR content may include pre-operative imaging, intra-operative imaging, instrument data, or procedure commands. Intra-operative imaging can be acquired via indocyanine green (ICG) fluorescence imaging. AR content may include real-time surgical data received from another connected system. AR content may include any combination of data derived from procedures for use, instrument settings, instrument commands for use, instrument status, operating parameters, detected irregularities, or instrument operation.

[0285] Figure 43 shows an exemplary flow of hub operation under a visualization control mode with AR capabilities. The hub can acquire AR control parameters for controlling multiple AR devices in 17701. These AR control parameters may include user roles, user orientation to the display, surgical procedure progress, surgical context, real-time user input, and / or pre-configured user preferences.

[0286] In step 17702, the hub may then determine different AR content to be overlaid on different AR devices based on AR control parameters. Based on the determined AR content for different AR devices, the hub may, in step 17703, transmit the respective AR content to the respective AR devices.

[0287] AR content may include instructions for use associated with a surgical instrument, device settings, device status, device commands for use, at least one operating parameter, or indication of detected anomalies.

[0288] The AR control parameter may be the user's orientation to the display, and different AR content for overlaying via different AR devices may be determined based on the user's orientation to the display. Figure 46 shows an exemplary flow of hub operation under a visibility control mode having AR capabilities that enable overlays on various displays. In 17730, the hub may acquire the AR control parameter. In 17731, the hub may determine overlay data for overlaying on the display via the AR device based on the AR control parameter. In 17732, the hub may detect the user of the AR device looking at the display. In 17733, the hub may overlay the overlay data on the content displayed on the display via the AR device. For example, if it is determined that the user is looking at the display, the hub may generate and overlay AR content associated with that display (e.g., AR content associated with the content displayed on the display). If it is determined that the user is not looking at the display, the hub may remove the AR content associated with the display from the AR device.

[0289] In the example, the AR control parameters may be the user role associated with the AR device. Different AR content for overlaying across different AR devices may be generated based on the user role associated with each AR device.

[0290] Figure 45 shows an exemplary flow of hub operation under a visualization control mode with role-based AR capabilities. The hub can identify a first user role associated with a first AR device at 17721. The hub can determine a first overlay dataset for a first AR content at 17722 based on the first user role. The hub can then identify a second user role associated with the first AR device at 17723. The hub can determine a second overlay dataset for a second AR content at 17724 based on the second user role.

[0291] For example, a surgical hub may identify a user role associated with an AR device and a display type associated with a display. Based on the display type and user role, the surgical hub may determine AR content to overlay on content displayed on the display via the AR device. The display type may be an instrument display located on a smart surgical instrument, a shared display in the operating room, or a personal display. The AR content may be adjusted based on the display type of the display on which the AR content can be superimposed. For example, if the display is a larger screen display, the AR content may be resized to fit the image on the shared display. If the display is a surgical instrument display, the AR content may be resized to accommodate a smaller screen. For example, if the display is a surgical instrument display, surgical information that may not fit on the surgical instrument display may be added to the AR content to make such information available to the user.

[0292] Figure 21 shows an augmented reality system that can be controlled by multiple users. As shown, the system may include various OR displays 17100, including an instrument 17100(a), a primary display 17100(b), other displays 17100(c), a surgical hub 17104, and a smart device 17100(d). The hub 17104 and AR device, worn by users 17120(A), 17120(B), and 17120(C), can superimpose a predefined set of overlay data layers 17110(a) to 17110(e) onto the various OR displays 17100(a) to 17100(e). Medical professional users 17120(A), (B), and (C) may each wear augmented reality devices such as AR displays, AR goggles, or safety glasses with HMDs as described herein. The surgical hub and / or AR device can control access to specific displays and overlay data layers.

[0293] The AR content displayed on the AR device may be generated based on the user's role, context-awareness-related data, and / or visualization control mode (such as a subscription layer). As illustrated, the AR device of user 17120(A) may receive overlays 17110(a) to 17110(e) based on user 17120(A)'s role, operating status, and / or system hierarchy level, while the AR device of user 17120(B) may receive only overlays 17110(b) to 17110(e). As shown in Figure 21, the subsets of overlay data layers received by the AR device of user 17120(A) and the AR device of user 17120(B) may be the same, but some of the overlay data layers received by the device may differ. An exemplary interactive set of overlay data layer 17130 may include preoperative imaging, intraoperative imaging, instrument data, procedure information, and / or data generated based on the above. For example, users 17120(A), 17120(B), and 17120(C) can access different sets of overlays based on their different roles and different processes in different situations.

[0294] Figure 44 shows an exemplary flow of hub operation under visualization control with AR capabilities. In 17711, the hub may acquire AR control parameters as described herein. In 17712, the hub may acquire a data stream from a surgical instrument for display on a display. The data stream may be, or include, video images of the surgical site in the patient. In 17713, the hub may determine, based on the AR control parameters, first AR content to be overlaid on the data stream displayed on the display via a first AR device. The first AR content may include steps for using the surgical instrument, device settings, device status, device commands for use, at least one operating parameter, or an indication of detected anomalies. In 17714, the hub may determine, based on the AR control parameters, second AR content to be overlaid on the data stream displayed on the display via a second AR device. In 17715, the hub may transmit the AR content to each AR device based on the determined AR content for each AR device for display.

[0295] In an exemplary visualization control mode that supports augmented reality content, the surgical hub can overlay surgical information onto an anatomical model on the display. For example, based on the determination that the user associated with the AR device is a surgeon, the surgical hub may transmit AR content to the AR device that includes visualizations of tumors, tumor margins, and possible emphysema. For example, based on the determination that the user associated with the AR device is a surgeon's assistant, the surgical hub may transmit AR content that includes surgical procedures requiring assistance, device settings, and / or device status.

[0296] Figure 28 shows an exemplary display 5020 that can be viewed from the AR device. Display 5020 includes screen content displayed on a screen overlaid with AR content. Display 5020 can depict an information index 5022 and a model of an anatomical structure 5024 generated by the control system of the surgical visualization system. The anatomical structure 5024 may include unaffected tissue 5026 that is neither diseased nor occupied by any significant structure. The model of the anatomical structure 5024 can depict detected and / or determined features such as the target tissue 5028, a predetermined margin 5030, an excision margin 5032, a first feature 5034 of the anatomical structure 5024, and an adjusted excision margin 5036. The control system 133 of the surgical visualization system has designated each of these detected features of the anatomical structure 5024 with a specific color, and Display 5020 can depict each of the detected features with its specific designated color, as shown by the cross-hatching in Figure 28. The information index 5022 can depict the correlation between each specific color and information associated with its designated detected feature. For example, the information index 5022 in Figure 28 correlates each specific color with a textual description of the corresponding feature of an anatomical structure 5024. In other embodiments, the information index 5022 correlates each specific color with additional information associated with the corresponding feature.

[0297] As depicted in Figure 28, the surgical visualization system can detect a target tissue 5028 within an anatomical structure 5024. The information index 5022 of the display 5020 can indicate that the detected target tissue 5028 is a tumor. Instructions stored in the memory of the control system of the surgical visualization system can instruct the control circuit to apply a predetermined margin 5030 around the target tissue 5028 based on the detected quality of the tumor, including its size, geometric shape, and / or type. Thus, the control system 133 can assign a specific color to the excision margin 5030, and the information index 5022 can correlate the specific color with additional information associated with the excision margin 5030. The control circuit of the surgical visualization system can determine an excision margin 5032 around the target tissue 5028, taking into account the detected target tissue 5028 and the predetermined margin 5030. In the display 5020 of Figure 28, the excision margin 5032 is depicted by line segments around the anatomical structure 5024 corresponding to the capabilities of the intended surgical instrument. For example, the surgical instrument may be a surgical stapler configured to staple the tissue before cutting it in a linear stroke. However, the indication 5020 may alternatively depict the excision margin 5032 if other surgical instruments are implemented.

[0298] Display 5020 in Figure 28 depicts a feature 5034 of an anatomical structure 5024 detected by the surgical visualization system. The information index 5022 of Display 5020 in Figure 28 may indicate that the detected feature 5034 of the anatomical structure 5024 is tissue 5026 damaged by emphysema. The AR content may include the initially determined excision margin 5032 in Figure 28, which can traverse the feature 5034 of the anatomical structure 5024. The control circuit of the surgical visualization system may determine an adjusted excision margin 5036 to encompass the feature 5036, the target tissue 5028, and a predetermined margin 5030. The AR content may include the adjusted excision margin 5036 via a dashed line. Such AR content may allow a clinician during surgery to select either the initially determined excision margin 5032 or the adjusted excision margin 5036. In another embodiment, the indicator 5020 restricts the clinician during surgery to a modified resection margin 5036 based on instructions stored in the control system's memory.

[0299] For example, AR content may be generated for surgical planning and / or detection of critical structures. Referring here to Figure 29, a three-dimensional model 5068 of an anatomical structure 5069 generated by a surgical visualization system is depicted. The surgical visualization system may include an imaging device 5070 having a distance sensor system 5071 having an emitter 5072 configured to emit electromagnetic radiation 5074 onto the anatomical structure 5069 and a receiver 5076 configured to detect the reflected electromagnetic radiation 5074. The imaging device 5070 in Figure 29 can utilize the aforementioned spectral light, structured light, and laser Doppler techniques to identify critical structures such as tumors 5078 and generate a fully integrated model 5068 and detailed characterization of the anatomical structure 5069. For example, the three-dimensional model 5068 in Figure 25 can depict the anatomical structure 5069 as the upper lobe of the right lung, and can specifically depict various features of the anatomical structure 5069, such as the artery 5080, vein 5082, bronchus 5084, upper lobe bronchus 5086, right pulmonary artery 5090, and / or main bronchus 5092. Although the anatomical structure 5069 in Figure 29 is the lung, the surgical visualization system can model various anatomical structures depending on the intended implementation. Thus, the surgical visualization system can use spectral light, structured light, and / or laser Doppler to characterize any anatomical structure and display the detected features in detail in the three-dimensional model.

[0300] The AR content may include proximity alerts when the distal apex of a surgical instrument moves within a specific range of a critical structure 5078. For example, real-time three-dimensional spatial tracking of the distal apex of a surgical instrument may be performed. The distance sensor system 5071 of the imaging device 5070 may be positioned on the distal tip of the surgical instrument. Thus, the emitter 5072 can emit electromagnetic radiation 5074 onto the surface of the anatomical structure 5069, and the receiver 5076 can detect the electromagnetic radiation 5074 reflected from the surface of the anatomical structure 5069. The surgical visualization system can determine the position of the emitter 5072 relative to the surface of the anatomical structure 5069 based on the time of flight of the electromagnetic radiation 5074, or the time between the emission from the emitter 5072 and its detection by the receiver 5076. The surgical visualization system uses a distance sensor system 5071 and time-of-flight techniques to determine the position of surgical instruments relative to an anatomical structure 5069, but other suitable components and / or techniques may be used to achieve the same effect, including the position of surgical instruments in a three-dimensional model 5068 of the anatomical structure 5069.

[0301] In the example, the AR control parameter may be the progress of the surgical procedure, and different AR content for overlaying via different AR devices may be determined based on the progress of the surgical procedure. For example, based on the surgical progress approaching a dissection, the AR content provided to the AR device associated with the surgeon may include the proposed dissection route. The AR content provided to another AR device may include a notification that the surgery is approaching a critical stage.

[0302] Referring here to Figure 30, a display of the three-dimensional model 5068 of Figure 29 is shown according to at least one aspect of the present disclosure. The AR content may include a resection margin overlay configured to depict a user-selected resection path 5096 and a system-proposed resection path 5104. For example, the resection margin overlay may further depict detected features such as arteries 5080, veins 5082, and bronchi 5084, detected target tissue such as a tumor 5094, and / or a predetermined margin 5095 based on instructions stored in memory 134 (Figure 2). A clinician in surgery may examine the AR content superimposed or overlaid on the surgical display to determine a user-selected resection path 5096 to remove the tumor 5094 and the predetermined margin 5095. For example, a clinician in surgery may determine a user-selected resection path 5096 that can optimize the remaining volume of an anatomical structure 5069, such as lung volume. Therefore, during surgery, the clinician can provide the surgical visualization system with a user-selected cutting path 5096 via the user interface.

[0303] The surgical visualization system can receive a user-selected cutting path 5096 via a user interface and evaluate the user-selected cutting path 5096 against the location of any detected feature of the anatomical structure 5069. For example, as depicted in Figure 30, the surgical visualization system can identify that the user-selected cutting path 5096 interferes with arteries 5080, veins 5082, and bronchi 5084 of the anatomical structure 5069. Thus, a combined view 5093 (e.g., a surgical display superimposed with AR content) can depict the anticipated interference and issue a notification to the clinician during surgery. The notification may be visual, audible, tactile, and / or any combination thereof. The display may further highlight features or portions of the anatomical structure 5069 affected by the user-selected cutting path 5096, and / or portions of the anatomical structure 5069 that may become inviolable due to the user-selected cutting path 5096. For example, the AR content may highlight a section 5098 of artery 5080 to represent a blood supply 5100 affected by a user-selected severance route 5096. The AR content may also highlight a portion 5102 of an anatomical structure 5069 that could become inviolable due to a lack of blood or air as a result of the user-selected severance route 5096.

[0304] Additionally and / or alternatively, the AR content may include a system-proposed cutting route 5104 that optimizes the remaining volume of an anatomical structure 5069, removes the target tissue 5094 and a predetermined margin 5095, and minimizes adverse effects on the detected features of the anatomical structure 5069. For example, the system-proposed cutting route 5104 may maintain a small remaining volume of the anatomical structure 5069 without interfering with arteries 5080, veins 5082, and bronchi 5084, and still remove the tumor 5094 and a predetermined margin 5095 from the upper lobe of the lung. In some embodiments, the surgical visualization system may allow the clinician during surgery to select either a user-selected cutting route 5096 or a system-proposed cutting route 5104. In other embodiments, the surgical visualization system may allow the clinician during surgery to reject the system-proposed cutting route 5104 and input a second user-selected cutting route based on the information depicted on the display.

[0305] Referring here to Figure 31, AR content is depicted in combination with a display view 5106 of a three-dimensional model 5108 of an anatomical structure 5110 generated by a surgical visualization system 5107 according to at least one aspect of the present disclosure. The surgical visualization system 5107 may include surgical instruments 5109 having a distance sensor system, a structured light system, a spectral light system, or any combination thereof. A clinician in surgery can examine the display 5106 to determine a user-selectable cutting path 5112 for removing target tissue from the anatomical structure 5110. The surgical visualization system 5107 in Figure 31 can receive the user-selectable cutting path 5112 via a user interface and evaluate the user-selectable cutting path 5112 against the location of any detected feature of the anatomical structure 5110. For example, the surgical visualization system 5107 in Figure 31 identifies that the user-selectable cutting path 5112 may interfere with a deflated portion 5114 of the anatomical structure 5110. Shrunken portions 5114 of the anatomical structure 5110 may negatively affect the excision of the target tissue and may lead to postoperative complications, including a smaller-than-optimal residual volume of the anatomical structure 5110. Therefore, AR content may include instructions for anticipated problems and notifications to the clinician during surgery. Notifications may be visual, audible, tactile, and / or any combination thereof.

[0306] Additionally and / or alternatively, the AR content shown in Figure 31 can depict a system-suggested cutting path 5116, which can be overlaid on the display. The system-suggested cutting path can optimize the remaining volume of the anatomical structure 5110, remove the target tissue and predetermined margins, and / or minimize adverse effects caused by detected features of the anatomical structure 5110. For example, cutting atrophied tissue 5114 can complicate the surgical procedure and introduce unnecessary risks. The system-suggested cutting path 5116 in Figure 31 minimizes this risk by instructing the clinician during surgery on the fully expanded tissue of the anatomical structure 5110. In some embodiments, the surgical visualization system 5107 may allow the clinician during surgery to select either a user-selected cutting path 5112 or a system-suggested cutting path 5116. In other embodiments, the surgical visualization system 5107 may allow the clinician during surgery to reject the system-suggested cutting path 5116 and input a second user-selected cutting path based on the information depicted on the display 5106.

[0307] The surgical instruments described herein may be configured to include a distance sensor system or other means that enable the surgical visualization system to detect the position of the surgical instrument relative to an anatomical structure. The surgical visualization system described herein may also notify the clinician during surgery by issuing a notification via the AR device if the detected position of the surgical instrument does not conform to a selected cutting path. The surgical visualization system may issue visual, audible, and / or tactile notifications to the clinician during surgery via the AR device indicating that the surgical instrument should be repositioned before commencing the surgical procedure. In some embodiments, the surgical visualization system may prevent the clinician during surgery from performing the surgical procedure via the AR device until the surgical instrument is properly positioned according to a selected cutting path depicted on the display.

[0308] The display of automatically adjustable tumor margins based on visually identifiable major structures, anomalies, and instrument-sensing tissue characteristics is further described in U.S. Patent Application No. 16 / 729,778, filed December 31, 2019, entitled “SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE” (Agent Reference Number: END9219USNP1), which is incorporated herein by reference in its entirety.

[0309] In the example, the AR content may include visualization of the obscured portion of the surgical site. The visualization of the obscured portion of the surgical site may be overlaid on a live stream of the surgical site in the operating room from a medical imaging device. The visualization of the obscured portion of the surgical site may be generated using a multispectral EMR source.

[0310] Figure 32 shows an exemplary fused image generated from a multispectral EMR source. A fused image can be generated using image data from at least three different EMR wavelength ranges to produce the resulting image. Multiple images can be used to collectively visualize a surgical site in corresponding EMR wavelength ranges. For example, the first image may be captured using the visible light portion of the EMR spectrum, including the first unobstructed portion and the rest of the image being obscured; the second image may be captured using the MWIR portion of the EMR spectrum, including the second unobstructed portion; and the third image 3042c may be captured using the LWIR portion of the EMR spectrum, including the third unobstructed portion. For example, the fourth image may be captured using the visible light portion of the EMR spectrum and thus correspond to the first image, but may include additional image processing to identify the portion obscured by fluid (water). Thus, the corresponding portion of the first image can be filtered by the corresponding wavelength or wavelength range (e.g., the blue-green portion of the visible light spectrum) to remove the obstruction.

[0311] The combined or fused image 3070 may be generated from the initial images described above. The fused image 3070 may include a first portion 3072 corresponding to the unobstructed portion of a first image generated from the visible light portion of the EMR spectrum, a second portion 3074 corresponding to the unobstructed portion of a second image generated from the MWIR portion of the EMR spectrum, a third portion 3076 corresponding to the unobstructed portion of a third image generated from the LWIR portion of the EMR spectrum, and a fourth portion 3078 corresponding to the obscured portion of an image generated from the visible light portion of the EMR spectrum, but post-processed to remove the blue-green portion of the visible light spectrum. Each of the aforementioned image portions 3072, 3074, 3076, and 3078 can be fused together to generate a fused image 3070 that provides unobstructed visualization of the tumor 3038 and any other relevant structures 3040.

[0312] The use of fused images is described in detail in U.S. Patent Application No. 16 / 729,807, filed December 31, 2019, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY" (Agent Reference Number: END9228USNP1), which is incorporated herein by reference in its entirety.

[0313] Figures 34A–34C illustrate an example of a series of surgical steps for the removal of an intestinal / colon tumor, and allow for the benefit of AR content generated using multi-image analysis of the surgical site. Figure 34A depicts a portion of the surgical site, including the intestine 2932 and branched vascular system 2934 that supply blood and nutrients to the intestine 2932. The intestine 2932 may have a tumor 2936 surrounded by a tumor margin 2937. A first optical sensor module of the visualization system may have a wide field of view 2930 and can provide imaging data of the wide field of view 2930 to the display system. A second optical sensor module of the visualization system may have a narrow field of view or a standard field of view 2940 and can provide imaging data of the narrow field of view 2940 to the display system. In some embodiments, the wide-field and narrow-field images may be displayed by the same display device. In other embodiments, the wide-field and narrow-field images may be displayed by separate display devices.

[0314] During surgical procedures, it may be important to remove not only the tumor 2936 but also its surrounding margin 2937 to ensure complete removal of the tumor. A wide-angle field of view 2930 may be used to image both the vascular structure 2934 and the portion of the intestine 2932 surrounding the tumor 2936 and margin 2637. As mentioned above, the vascular structure supplying the tumor 2936 and margin 2637 should be removed, but the vascular structure supplying the surrounding intestinal tissue must be preserved to provide oxygen and nutrients to the surrounding tissue. The migration of vascular structures supplying the surrounding colon tissue deprives the tissue of oxygen and nutrients, leading to necrosis. In some cases, laser Doppler imaging of the tissue visualized within the wide-angle field of view 2630 can be analyzed to provide speckle contrast analysis 2933 showing blood flow within the intestinal tissue.

[0315] AR content may include indications of blood flow within the tissue. For example, AR content may include indications of which parts of the vascular tree can supply blood to the tumor. Figure 34B shows the procedure during a surgical operation. The surgeon may be uncertain which parts of the vascular tree supply blood to the tumor 2936. The surgeon may test a vessel 2944 to determine whether it supplies blood to the tumor 2936 or healthy tissue. The surgeon may clamp the vessel 2944 using a clamping device 2812 and determine the portion of intestinal tissue 2943 that is no longer perfused by speckle contrast analysis. A narrow field of view 2940 displayed on the imaging device can assist the surgeon in the magnification and detail work required to visualize the single vessel 2944 being tested. Once the suspected vessel 2944 is clamped, a portion of the intestinal tissue 2943 is determined to be devoid of perfusion based on Doppler imaging speckle contrast analysis. The suspected vessel 2944 does not supply blood to the tumor 2935 or the tumor rim 2937, and is therefore recognized as a reserve vessel during surgical procedures.

[0316] Figure 34C depicts the next stage of the surgical procedure. In this stage, a supply vessel 2984 is identified to supply blood to the edge 2937 of the tumor. When this supply vessel 2984 is severed, blood is no longer supplied to the portion of the intestine 2987 that may include at least a portion of the edge 2937 of the tumor 2936. In some embodiments, the lack of perfusion to the portion of the intestine 2987 can be determined by speckle contrast analysis based on Doppler analysis of blood flow into the intestine. The unperfused portion of the intestine 2987 may then be isolated by a seal 2985 applied to the intestine. In this way, only the vessels perfusing the tissue to be removed surgically are identified and sealed, thereby preserving healthy tissue from unintended surgical consequences.

[0317] AR content may be generated based on imaging analysis of the surgical site. The surgical site may be examined for the effectiveness of the surgical manipulation of the tissue. Non-limiting examples of such examinations may include examination of surgical staples or welds used to seal the tissue at the surgical site. Cone beam coherence tomography using one or more illumination sources may be used in such a method. AR content may include landmarks indicated within the image of the surgical site. In some examples, landmarks may be determined by image analysis techniques. In some examples, landmarks may be indicated by manual intervention of the image by the surgeon. In some embodiments, non-smart-ready visualization methods may be imported for use in hub image fusion techniques.

[0318] Instruments not integrated into the hub system may be identified and tracked during use within the surgical site. In this embodiment, the computing and / or storage components of the hub (including, for example, a cloud system) may include a database of identifiable images of EES and competing surgical instruments from one or more images acquired by any image acquisition system or visual analysis of such alternative instruments. Image analysis of such instruments may further enable identification when instruments are swapped with different instruments to perform the same or similar jobs. Identification of instrument swaps during surgical procedures may provide relevant information when an instrument does not perform the job of the instrument or when it malfunctions.

[0319] In the example, the AR content may include anatomical identification information that can be generated based on preoperative images. The AR content may be overlaid on video images of the surgical site within the patient. The anatomical identification information may be overlaid on a live stream of the surgical site in the operating room from a medical imaging device.

[0320] Figure 35 shows an example of an augmented video image 6350 that includes a preoperative video image 6352 augmented with data 6354, 6356, and 6358 that identify the displayed elements. AR data can be overlaid or superimposed on the preoperative image 6352 via an AR device. A preoperative image 6352 of an anatomical cross-section of the patient can be generated. An augmented video image of the surgical site within the patient may be generated. The augmented video image 6350 may include images of at least some of the surgical tools 6354 manipulated by the user 6456. The preoperative image 6352 may be processed to generate data about the anatomical cross-section of the patient. The AR content may include labels 6358 of the anatomical cross-section and at least some of the peripheral margins of the anatomical cross-section. The peripheral margins can guide the surgeon to the cutting position relative to the anatomical cross-section, embed data and user 6356 identification information within the preoperative image 6350, and be configured to display the augmented video image 6350 about the anatomical cross-section of the patient to the user. The system can sense the load state on the surgical tool 6354 and generate a feedback signal based on the sensed load state. AR content, including user identification data and location information of the user operating the surgical tool 6354, can be updated in real time in response to changes in the position of the surgical tool 6354 within the augmented video image 6350. Further examples are disclosed in U.S. Patent No. 9,123,155, issued September 1, 2015, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES," which is incorporated herein by reference in its entirety.

[0321] Using radiographic integration technology, preoperative images 6352 can be overlaid with data obtained by live internal detection or pre-procedure technology. Radiographic integration may include marker and landmark identification using the identification of surgical landmarks, radiographic markers placed inside and outside the patient, radiopaque staples, clips, or other tissue fixation items. Digital radiographic technology can be used to generate digital images for overlaying with preoperative images 6352. Digital radiographic imaging is a form of X-ray imaging that uses a digital image capture device with a digital X-ray sensor instead of conventional photographic film. Digital radiographic technology can provide immediate image preview and usability for overlaying with preoperative images 6352. Furthermore, special image processing techniques can be applied to digital X-ray images to improve the overall display quality of the images.

[0322] Digital radiation technology can utilize imaging detectors, including flat panel detectors (FPDs), which are classified into two main categories: indirect FPDs and direct FPDs. Indirect FPDs contain amorphous silicon (a-Si) combined with a scintillator in the outer layer of the detector, which is made from cesium iodide (CSI) or gadolinium oxy-sulfide (Gd2O2S) to convert X-rays into light. The light can pass through the a-Si photodiode layer, where it is converted into a digital output signal. The digital signal is then read out by a thin-film transistor (TFT) or a fiber-coupled charge-coupled device (COD). Direct FPDs include amorphous selenium (a-Se) FPDs, which directly convert X-ray photons into charges. In this design, the outer layer of the flat panel is typically a high-voltage bias electrode. X-ray photons generate electron-hole pairs within the a-Se layer, and the passage of these electrons and holes depends on the potential of the bias voltage charge. Once the holes are replaced by electrons, the resulting charge pattern in the selenium layer is read out by a TFT array, active matrix array, electrometer probe, or microplasma line addressing. Other direct digital detectors are based on CMOS and CCD technology. Phosphor detectors may also be used to record X-ray energy during exposure, which is scanned by a laser diode to excite the stored energy that is emitted and read out by a digital image capture array on a CCD.

[0323] In this example, the AR control parameters may be real-time user input, and different AR content to be overlaid across different AR devices may be determined based on user input. For example, the user interface may present the user with a selection of one or more AR content to display on the AR device. The hub can generate and transmit AR content according to the user selection.

[0324] Figure 36 shows an example of a customizable AR content overlay. As illustrated, the AR content option 17410 may be presented, for example, on an interactive display screen. The AR content option 17410 may include an available overlay layer that can include preoperative tumor MRI, other relevant preoperative data, ICG data, real-time Doppler monitoring, procedure steps, device status, and other overlays customizable by the user. The overlay layer may be provided by a hub. In this example, preoperative tumor data and real-time Doppler monitoring are selected, and such data is included in the AR content that is overlaid on the surgical image. Through the AR device, the user can see vision 17420 showing two selected overlays, namely preoperative tumor MRI and real-time Doppler monitoring. As illustrated, the AR content may include markings of the tumor 14733 and tumor margin 17432. With the help of the overlay, the user can clamp jaw 17436 onto a vessel to verify whether the vessel is inside or outside the tumor margin. The AR content may show blood flow in the vessel. For example, whether a blood vessel is associated with low or high blood flow may be indicated through color coding in the AR content. For illustrative purposes, the AR content may include changing low-flow blood vessel 17434 to a blue blood vessel and high-flow blood vessel 17438 to a red blood vessel.

[0325] A hub communicating with an augmented reality device can provide simulation or confirmation of an intended action. AR content may include instructions for predicting results if the user performs the intended action. For example, if a user clamps or holds jaws over an intended area to staple, cut, or seal, the AR content may show the user changes in fluid flow. This may provide the user with guidance for moving in one direction or another. For example, a surgical hub may receive instructions for an intended action on a target area. The instructions may include images captured through a surgical scope showing that a surgical instrument is positioned on or near the target area. The instructions may also include images captured through an AR device showing that a surgical instrument is positioned on or near the target area. For example, AR content may be generated based on microwave ablation confirmation, which can show predictive output based on time and temperature. The surgical hub can receive visual input from cameras in the OR and sensor input from surgical devices in the OR. The surgical hub can combine and compile incoming inputs to generate confirmation and / or feedback of expected results for inclusion in AR content. The hub can synthesize various data streams into a coherent output, which may be overlaid or displayed on displays including primary and / or secondary displays, AR displays, and / or non-AR displays. The surgical hub may also obtain predictive results associated with performing intended actions on a target region and may include these predictive results in AR content. The predictive results may be determined based on visual data received from a surgical scope and surgical data received from surgical instruments. The predictive results may be determined by the surgical hub or with the help of a remote server. For example, the surgical hub may obtain visual data from a surgical scope, obtain sensor input data from at least one surgical instrument, and send the visual and sensor input data to a remote server.The prediction results are received from a remote server and may be included in AR content for display on an AR device.

[0326] Figure 44 shows an exemplary flow of hub operation under visualization control with AR capabilities. The AR content to be overlaid on the display may vary depending on the AR device. In 17711, the hub may acquire AR control parameters as described herein. The AR control parameters may include at least one of the following: user role, user orientation to the first display, progress of surgical procedure, surgical context, real-time use input, or pre-configured user preferences. In 17712, the hub may acquire a data stream from a surgical instrument for display on the display. The data stream may be, or include, video images of the surgical site in the patient. In 17713, the hub may determine, based on the AR control parameters, AR content to be overlaid on the data stream displayed on the display via the first AR device. An AR device for use by a surgeon may display different AR content than the AR content displayed via an AR device for use by a surgical assistant. An AR device with one pre-configured user preference may display different AR content than the AR content displayed via an AR device with different pre-configured user preferences. AR content may include instructions for use associated with a surgical instrument, instrument settings, instrument status, instrument commands for use, operating parameters, and / or indications of detected anomalies. In 17714, the hub may determine AR content to be overlaid on a data stream displayed on a display via a second AR device, based on AR control parameters. In 17715, the hub may transmit AR content to each AR device based on the determined AR content for each AR device for display.

[0327] Examples of disclosures 1. A surgical hub, A communication array configured to be operably connected to at least one display, multiple AR (augmented reality) devices, and at least one surgical instrument, A processor, and the processor, To obtain AR control parameters, Obtaining a first data stream from at least one surgical instrument for display on a first display of at least one display, Based on AR control parameters, determine a first AR content to be overlaid on a first data stream displayed on a first display via a first AR device among multiple AR devices, Based on AR control parameters, determine a second AR content to be overlaid on a first data stream displayed on a first display via a second AR device among multiple AR devices, wherein the second AR content is different from the first AR content. A surgical hub configured to transmit a first AR content and a second AR content to a first AR device and a second AR device, respectively.

[0328] For example, in Example 1, the first display may be a primary display or a secondary display. Obtaining AR control parameters may include determining, measuring, or receiving AR control parameters. The processor may determine the first and / or second AR content based on whether the AR control parameters exceed a threshold.

[0329] For example, in Example 1, the processor may be further configured to transmit a first data stream to a first display. The processor may be further configured to overlay first AR content on the first data stream via a first AR device and to overlay second AR content on the first data stream via a second AR device.

[0330] For example, in Example 1, the first and / or second AR device may be safety glasses, augmented reality goggles, or a head-mounted display having an augmented reality display. Surgical instruments may include laparoscopic instruments, endoscopic instruments, laparoscopes, electrosurgical instruments, ultrasonic surgical instruments, and / or surgical stapling instruments. The first data stream may be, or may include, video images of the surgical site in the patient. The first display may include, or may include, an instrument display mounted on a surgical instrument such as at least one surgical instrument, an operating room or in-operating room display, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, or a wrist-mounted display.

[0331] For example, in Example 1, the surgical hub may include an input device configured to receive inputs and transmit them to a processor. The input device may include an interactive display. The first display may include an input device. The input device may allow AR control parameters, user roles or intended users for the first and / or second AR devices, and / or display control indicators to be input and transmitted to the processor.

[0332] 2. AR control parameters include user roles, and the processor, Identifying a first user role associated with a first AR device, Based on the first user role, determine the first overlay dataset to be included in the first AR content, Identifying a second user role associated with a second AR device, The surgical hub of Example 1 is further configured to determine, based on a second user role, a second overlay dataset to be included in a second AR content, wherein the second overlay dataset is different from the first overlay dataset.

[0333] For example, in Example 2, identifying a first user role associated with a first AR device may include identifying, selecting, or receiving data that identifies the intended user role for the first AR device. Identifying a second user role associated with a second AR device may include identifying, selecting, or receiving data that identifies the intended user role for the second AR device. Users and user roles may include surgeons, surgical assistants, and / or medical professionals who may be located inside or outside a sterile field. Determining a first overlay dataset based on a first user role may include generating a first overlay dataset for inclusion in the first AR content of data intended for display to the intended user role of the first AR device. Determining a second overlay dataset based on a second user role may include generating a second overlay dataset for inclusion in the second AR content of data intended for display to the intended user role of the second AR device.

[0334] 3. The processor, Detecting that the user of the first AR device is looking at the first display, A surgical hub, example 1 or 2, is further configured to instruct the first AR device to overlay first AR content onto a first data stream displayed on a first display via the first AR device in response to detection.

[0335] 4. The first AR content is, Process for the use of surgical instruments, Device settings, Device status, Device instructions for use, Operating parameters, or A surgical hub from any of Examples 1-3, including at least one of the indicators of detected abnormalities.

[0336] 5. A surgical hub, one of Examples 1-4, in which the first AR content includes at least one of pre-operative imaging, intra-operative imaging, instrument data, or procedure instructions.

[0337] For example, in any one of Examples 1-5, the first and / or second AR content includes: steps for using a surgical instrument, device settings, device status, device commands for use, operating parameters, indicators of detected anomalies, preoperative imaging, intraoperative imaging, instrument data, procedure commands, indicators of predicted results if the user performs the intended action, changes in fluid flow, intravascular blood flow such as low or high blood flow, indicators of which blood vessels supply blood to the tumor, tumor marking, tumor margin, resection margin, resection or tumor margin adjusted during surgical procedure, real-time Doppler monitoring, preoperative MRI data, indocyanine green (ICG) fluorescence imaging data, surgical The system may include at least one of the following: user identification information for operating the tool; real-time surgical data received from another connected system; proximity alerts when a surgical instrument, such as at least one surgical instrument (or its distal apex), moves within a specific range of a critical structure; the portion of a blood vessel to represent the blood supply affected by the cutting path; a portion of an anatomical structure that could become inviolable due to lack of blood or air due to the cutting path; a hub-proposed cutting path or a processor-proposed cutting path; indicators of anticipated problems; visualization of obstructed portions of the surgical site; landmarks shown in images of the surgical site; and / or anatomical identification information generated based on preoperative images.

[0338] 6. AR control parameters are, User orientation relative to the first display, Progress of surgical procedure, Surgical context, Real-time user input, or A surgical hub, one of Examples 1-5, which includes at least one of the pre-configured user preferences.

[0339] For example, in any one of Examples 1 to 6, the AR control parameters may include at least one of the following: user orientation to the first display, user orientation to the first display, user head orientation to the first display, whether the user is looking at the first display, progress of the surgical procedure, surgical context, real-time user input, user roles associated with the first and / or second AR devices, or intended user roles for the first and / or second AR devices, display type of the first display, and / or pre-configured user preferences.

[0340] 7. The first AR content to be overlaid on the first data stream displayed on the first display is further determined based on the display type associated with the first display, and the processor, One of the surgical hubs, Examples 1-6, is further configured to determine a third AR content to be overlaid on content displayed on the second display via a first AR device, based on the display type associated with the second display of at least one of the displays.

[0341] For example, in any one of Examples 1 to 7, the first AR content to be overlaid on a first data stream displayed on a first display may be further determined based on the display type of the first display, and the processor is further configured to determine a third AR content to be overlaid on content displayed on a second display via a first AR device, based on the display type of a second display among at least one of the displays.

[0342] 8. Display type is Instrument displays placed on smart surgical instruments, or A shared display in the operating room, or A surgical hub, example 7, including at least one of the following: a personal display.

[0343] For example, in any one of Examples 1 to 8, the display type of the first and / or second display may include an instrument display mounted on a surgical instrument such as at least one surgical instrument, a display in or within an operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, or a wrist-worn display.

[0344] 9. The processor, Receiving display control indicators from the first AR device, A surgical hub, one of Examples 1-8, is further configured to adjust content for display on a first display based on received display control indicators.

[0345] The display control indicator may be a selection of display content. The surgical hub may include an input device configured to receive and transmit the display control indicator to a processor. The input device may include an interactive display. The first display may include an input device.

[0346] 10. The processor, The first AR device detects whether it is outside the boundary of the surgical room, One of the surgical hubs in Examples 1-9 is further configured to disable the transmission of the first AR content to the first AR device in response to detecting that the first AR device is outside the boundary of the surgical room.

[0347] 11. The processor, Receiving indicators of intended actions in the target area, Obtaining predictive results associated with performing intended actions on the target region, One of the surgical hubs, Examples 1-10, is further configured to include the prediction results in the first AR content.

[0348] For example, in Example 11, obtaining a prediction result may include determining the prediction result. The intended action on the target area may include at least one of the following: a surgical procedure performed on the target tissue, a procedure for using a surgical instrument, a clamping operation, a cutting operation, a stapling operation, a transverse incision operation, and / or the application of ultrasound and / or RF energy.

[0349] The indicator may include images acquired via a surgical scope such as a laparoscope or endoscope and / or surgical instrument and / or a first AR device, showing that the surgical instrument is positioned on or near a target area, visual input from a camera in the operating room, and sensor input from a surgical device such as a surgical instrument.

[0350] The prediction results may be determined based on visual data received from a surgical scope and / or surgical data received from at least one surgical instrument, such as a surgical instrument. The prediction results may be obtained and / or determined by a processor in the surgical hub, or by a remote server.

[0351] The surgical hub may be equipped with a situational awareness system, and the processor may use the situational awareness system to infer indicators of intended actions on the target area.

[0352] 12. A surgical hub of Example 11, in which the indicator includes a visualization captured via a first AR device or surgical scope, the visualization indicating that a surgical instrument is positioned on or near a target area.

[0353] 13. The processor, A surgical hub, example 11 or 12, further configured to determine a predictive outcome based on visual data received from a surgical scope and surgical data received from at least one surgical instrument.

[0354] For example, in Example 13, at least one surgical instrument may include a surgical scope. Obtaining or determining a prediction result may include determining the prediction result based on visual data received from the surgical scope and surgical data received from at least one surgical instrument.

[0355] 14. The processor, Acquire visual data from a surgical scope and sensor input data from at least one surgical instrument, The process involves sending visual data and sensor input data to a remote server, A surgical hub, example 11 or 12, further configured to receive prediction results from a remote server.

[0356] For example, in any one of Examples 11-14, receiving an index may include obtaining visual data from a surgical scope and / or sensor input data from at least one surgical instrument, transmitting the visual data and sensor input data to a remote server, and receiving prediction results from the remote server.

[0357] 15. A surgical hub, one of Examples 1-14, in which the first AR content includes at least one of visual or audible content.

[0358] For example, in any one of Examples 11 to 15, the first AR content includes at least one of visual content, audible content, and / or haptic content.

[0359] 16. A surgical hub, A communication array configured to be operably connected to at least one display, an AR (augmented reality) device, and at least one surgical instrument, A processor, and the processor, Obtaining a data stream from at least one surgical instrument for display on at least one display, To generate AR content to overlay on the display, Determining whether the user of the AR device is looking at the display, A surgical hub configured to detect when a user of an AR device is looking at the display, and to overlay AR content onto the content displayed on the display via the AR device.

[0360] For example, in Example 16, the processor may be further configured to send a data stream to a display and / or display the data stream on the display. Overlaying AR content on content displayed on the display via an AR device may include overlaying AR content on a data stream displayed on the display via an AR device....

Claims

1. It is a surgical hub, A communication array configured to be operably connected to at least one display, multiple AR (augmented reality) devices, and at least one surgical instrument, A processor and a processor, the processor To obtain AR control parameters, Obtaining a first data stream from one of the at least one surgical instruments for display on a first display of the at least one display, Based on the AR control parameters, determine a first AR content to be overlaid on the first data stream displayed on the first display via a first AR device among the plurality of AR devices, wherein the first AR content is obtained from one of the at least one surgical instrument. Based on the AR control parameters, determine a second AR content to be overlaid on the first data stream displayed on the first display via a second AR device among the plurality of AR devices, wherein the second AR content is obtained from one of the at least one surgical instruments, and the second AR content is different from the first AR content. A surgical hub configured to transmit the first AR content and the second AR content to the first AR device and the second AR device, respectively.

2. The AR control parameters include user roles associated with each AR device, and the processor, Identifying a first user role associated with the first AR device, Based on the first user role, a first overlay dataset to be included in the first AR content is determined, Identifying a second user role associated with the second AR device, The surgical hub according to claim 1, further configured to determine, based on the second user role, a second overlay dataset to be included in the second AR content, wherein the second overlay dataset is different from the first overlay dataset.

3. The first AR device further comprises a sensor, The sensor is configured to generate first orientation data for the first AR device relative to the first display. The AR control parameters include the first orientation data, The aforementioned processor, Based on the first orientation data, it is detected that the user of the first AR device is looking at the first display, The surgical hub according to claim 1 or 2, further configured to instruct the first AR device to overlay the first AR content on the first data stream displayed on the first display via the first AR device in response to the detection.

4. The first AR content described above is Process for the use of surgical instruments, Device settings, Device status, Device instructions for use, Operating parameters, or A surgical hub according to any one of claims 1 to 3, comprising at least one of the indicators of detected abnormalities.

5. The surgical hub according to any one of claims 1 to 4, wherein the first AR content includes at least one of pre-operative imaging, intra-operative imaging, instrument data, or treatment commands.

6. The aforementioned AR control parameters are The orientation of the user's head relative to the first display, Progress of surgical procedure, Surgical context, Real-time user input, or A surgical hub according to any one of claims 1 to 5, comprising at least one of pre-configured user preferences.

7. The first AR content to be overlaid on the first data stream displayed on the first display is further determined based on the display type associated with the first display, and the processor, The surgical hub according to any one of claims 1 to 6, further configured to determine a third AR content to be overlaid on content displayed on the second display via the first AR device, based on a display type associated with a second display among the at least one of the displays.

8. The aforementioned display type is An instrument display positioned on a smart surgical instrument operably connected to the aforementioned communication array, or A shared display in the operating room, or The surgical hub according to claim 7, comprising at least one of the following: a personal display.

9. The aforementioned processor, Receiving a display control indicator from the first AR device, The surgical hub according to any one of claims 1 to 8, further configured to adjust the first AR content for display on the first display based on the received display control indicators.

10. The communication array is further configured to be connected to a camera for capturing images and / or videos in a surgical operating room, The aforementioned processor, To acquire the aforementioned images and / or the aforementioned video, Based on the aforementioned images and / or video, the first AR device is detected to be outside the boundary of the surgical operating room. The surgical hub according to any one of claims 1 to 9, further configured to disable transmission of the first AR content to the first AR device in response to detection that the first AR device is outside the boundary of the surgical room.

11. The aforementioned processor, Receiving indicators of intended actions in the target area, Obtaining prediction results associated with performing the intended action on the target region, A surgical hub according to any one of claims 1 to 10, further configured to include the prediction results in the first AR content.

12. The surgical hub according to claim 11, wherein the indicator includes a visualization captured via a surgical scope operably connected to the first AR device or the communication array, and the visualization indicates that a surgical instrument is positioned on or near a target area.

13. The aforementioned processor, The surgical hub according to claim 11 or 12, further configured to determine the prediction result based on visual data received from a surgical scope operably connected to the communication array and surgical data received from the at least one surgical instrument.

14. The aforementioned processor, The system acquires visual data from a surgical scope operably connected to the communication array, and sensor input data from at least one surgical instrument. The visual data and the sensor input data are transmitted to a remote server. The surgical hub according to claim 11 or 12, further configured to receive the prediction results from the remote server.

15. The surgical hub according to any one of claims 1 to 14, wherein the first AR content includes at least one of visual content or audible content.

16. It is a surgical hub, A communication array configured to be operably connected to at least one display, an AR (augmented reality) device, and at least one surgical instrument, Equipped with a processor, The AR device includes a sensor, and the sensor is configured to generate first orientation data for the AR device relative to one of the at least one displays. The aforementioned processor, AR control parameters, which include the first orientation data, are obtained. To obtain content to be displayed on the display from one of the at least one surgical instruments, Based on the AR control parameters and data obtained from one of the at least one surgical instruments, AR content is generated to be overlaid on the display. Based on the AR control parameters, it is determined whether the user of the AR device is looking at the display. A surgical hub is configured to detect when the user of the AR device is looking at the display, and to overlay the AR content displayed on the display via the AR device.

17. The display is the first display of the at least one display, the sensor is configured to generate second orientation data for the AR device relative to the second display of the at least one display, the AR control parameters include the second orientation data, and the processor is configured The method involves generating a second AR content to be overlaid on the second display, wherein the second AR content is different from the AR content to be overlaid on the first display. Based on the second orientation data, it is determined whether the user of the AR device is looking at the second display, The surgical hub according to claim 16, further configured to detect that the user of the AR device is looking at the second display, and to overlay the second AR content onto the content displayed on the second display via the AR device.

18. The aforementioned processor, The surgical hub according to claim 17, further configured to stop supplying the AR content associated with the display via the AR device when it detects, based on the first and second orientation data, that the user of the AR device is no longer looking at the display.

19. The aforementioned processor, Receiving display control indicators from the AR device, A surgical hub according to any one of claims 16 to 18, further configured to adjust the content for display on the display based on the received display control indicators.

20. The AR control parameters include a user role associated with the AR device, The aforementioned processor, The surgical hub according to any one of claims 16 to 19, further configured to identify the user role, wherein the AR content is generated based on the user role.

Images