Shared contextual awareness of device actuator activities to prioritize specific aspects of displayed information.

Abstract

A surgical hub may be provided for prioritizing data on a display using situational awareness of the medical instrument. A first surgical task using the medical instrument during a medical procedure may be determined based on the context data. A second surgical task using the medical instrument may be determined based on the first surgical task and the context data. A message may be transmitted that may instruct the display to prioritize displayed data associated with the second surgical task.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application is related to the following, the contents of each of which are incorporated herein by reference. ·U.S. Patent Application titled "METHOD FOR OPERATING TIERED OPERATION MODES IN A SURGICAL SYSTEM" filed together with this specification and having Attorney Docket No. END9287USNP1, ·U.S. Patent Application titled "SITUATIONAL AWARENESS OF INSTRUMENTS LOCATION AND INDIVIDUALIZATION OF USERS TO CONTROL DISPLAYS" filed together with this specification and having Attorney Docket No. END9288USNP1, ·U.S. Patent Application titled "MONITORING OF USER VISUAL GAZE TO CONTROL WHICH DISPLAY SYSTEM DISPLAYS THE PRIMARY INFORMATION" filed together with this application and having Attorney Docket No. END9288USNP3, ·U.S. Patent Application titled "RECONFIGURATION OF DISPLAY SHARING" filed together with this specification and having Attorney Docket No. END9288USNP4, and ·U.S. Patent Application titled "CONTROL A DISPLAY OUTSIDE THE STERILE FIELD FROM A DEVICE WITHIN THE STERILE FIELD" filed together with this specification and having Attorney Docket No. END9288USNP5.

Background Art

[0002] Surgical systems often incorporate imaging systems that allow clinicians to view the surgical site and / or one or more parts thereof on one or more displays, such as monitors. These displays may be localized in the surgical theater and / or remote. The imaging system may include a scope equipped with a camera that views the surgical site and transmits the view to a display visible to the clinician. Examples of scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and exoscopy. The imaging system may be limited by the information that can be recognized by and / or communicated to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognizable during surgery by certain imaging systems. Additionally, certain imaging systems may not be able to communicate and / or transmit certain information to the clinician during surgery. [Overview of the Initiative] [Means for solving the problem]

[0003] According to one embodiment of the present invention, a surgical hub and / or medical instrument is provided for prioritizing data on a display. The surgical hub and / or medical instrument may include a processor. The processor may be configured to perform several actions. A surgical procedure may be determined. A first surgical operation using the medical instrument during the surgical procedure may be determined based on contextual data. A second surgical operation using the medical instrument may be determined based on the first surgical operation and contextual data. A message may be sent that can instruct the display to prioritize display data associated with the second surgical operation. The message may be a first message, and a second message may be sent to the medical instrument to instruct the medical instrument to configure according to the second surgical operation.

[0004] According to one embodiment of the present invention, a surgical hub and / or medical instrument is provided for prioritizing data on a display. The surgical hub and / or medical instrument may include a processor. The processor may be configured to perform several actions. A first surgical operation using the medical instrument during a surgical procedure may be determined based on contextual data. Instrument data may be received from the medical instrument and may be associated with the first surgical operation. A second surgical operation using the medical instrument can be determined based on the first surgical operation, instrument data, and the surgical procedure. A message can be sent that instructs the display to prioritize display data associated with the second surgical operation.

[0005] According to one embodiment of the present invention, a surgical hub and / or medical instrument is provided for prioritizing data on a display. The surgical hub and / or medical instrument may include a processor. The processor may be configured to perform several actions. A first surgical operation using the medical instrument during a surgical procedure may be determined based on contextual data. Instrument data may be received from the medical instrument and may be associated with the first surgical operation. Errors may be determined by analyzing instrument data from the medical instrument using contextual data. A second surgical operation using the medical instrument may be determined based on the first surgical operation, instrument data, and surgical procedure. A message can be sent that can instruct the display to prioritize display data associated with the second surgical operation. The display data may indicate errors.

[0006] According to one embodiment of the present invention, a surgical hub and / or medical device is provided for prioritizing data on a display. The medical device may include a display. Contextual data may be determined. A surgical procedure may be determined. A surgical operation in which the medical device is used during a surgical procedure may be determined based on the contextual data. Display data may be determined. The display data may be associated with a surgical operation, or with a user who may perform the surgical operation using the medical device. A message may be sent. The message may instruct the display to prioritize display data related to the surgical operation.

[0007] As an example above and below, surgical hubs and medical instruments can determine, display, and prioritize display data for the next surgical operation to help users, such as surgeons, perform a series of surgical steps. Surgical hubs and instruments have situational awareness of the operating room and can infer surgical operations, surgical procedures, and other data based on that awareness. This allows upcoming tasks to be determined, and data associated with these tasks to be generated, displayed, and prioritized for the user. This allows the user to be better prepared for the next surgical operation, which improves the efficiency of procedures and patient outcomes, reduces the risk of errors, and thereby improves patient safety. Data can be adjusted based on the current user of the instrument to further ensure that the user is prepared for upcoming tasks, and critical or dangerous tasks can be identified and highlighted in advance, allowing the user to be better prepared and improve their focus. This further improves efficiency and safety. Errors can be determined by analyzing contextual data, which allows the user to be quickly alerted when something goes wrong through data display and prioritization, enabling faster error correction. Furthermore, based on the specific nature of the error, commands to resolve the error, such as cleaning, reloading, or repairing, can be generated and provided in real time. This further improves error correction and enhances patient safety by resolving errors more effectively and quickly. Overall, users are better prepared for the surgical tasks they are performing, which leads to improved efficiency, patient outcomes, and patient safety.

[0008] Further embodiments of the present invention provide the following examples. 1. A surgical hub for prioritizing data on a display, wherein the surgical hub is Equipped with a processor, the processor, Based on contextual data, determine the first surgical operation in which medical instruments will be used during the medical procedure, Based on the first surgical procedure and contextual data, determine the second surgical procedure that involves the use of medical instruments. A surgical hub is configured to send messages to the display instructing it to prioritize display data associated with a second surgical operation.

[0009] For example, in Example 1, the surgical hub may use medical device context recognition to prioritize data on the display.

[0010] For example, in Example 1, the surgical hub may be configured to connect to a medical instrument and a display.

[0011] For example, in Example 1, determining the first surgical procedure may include determining, based on contextual data, that the first surgical procedure is currently being performed, and the first surgical procedure involves the use of medical instruments during a medical procedure.

[0012] For example, in Example 1, the medical device may be at least one of the following: a laparoscope, an endoscope, a thoracoscope, a laparoscopic surgical instrument, an electrosurgical instrument, an ultrasonic surgical instrument, and / or a surgical staple fastener.

[0013] For example, in Embodiment 1, the display may be an instrument display located on a medical instrument such as a medical device, a display in or within an operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, a wrist-worn display, an AR device, and / or a head-mounted display. The display may be configured to display data including display data. Sending a message instructing the display to prioritize display data associated with a second surgical operation may include sending a message instructing the display to prioritize display data associated with a second surgical operation over other display data.

[0014] For example, in Embodiment 1, sending a message instructing the display to prioritize display data associated with the second surgical operation may include making the display data larger, highlighting the display data, changing the color of the display data, moving the display data to a more prominent position, making non-display data semi-transparent, changing the color of non-display data, making non-display data smaller, and / or moving non-display data to a less prominent part of the display.

[0015] For example, in Embodiment 1, the displayed data may be associated with the second surgical operation by being related to the second surgical operation, being a procedural command for performing the second surgical operation, being patient data and / or instrument data related to the second surgical operation, being an indication that the second task was performed incorrectly or that an error occurred in the second task, being one or more commands for resolving an error in the second task, or being one or more commands for assisting the user when performing a corrective surgical operation.

[0016] For example, in Embodiment 1, the surgical hub may include a context-aware system configured to collect context-aware data and determine information from the context-aware data. For example, the context-aware system may be configured to determine or measure information such as context data, surgical operations such as a first surgical operation and / or a second surgical operation, medical procedures such as medical procedures, and / or user and / or user identification information. The context-aware system may include or be configured to connect at least one piece of hardware to transmit context-aware data to the context-aware system. The hardware may include at least one of the following: a camera, an in-operating room sensor, a user-associated sensor, a sensor associated with or part of a medical instrument, and / or a wearable device.

[0017] 2. The surgical hub according to Example 1, wherein the processor is further configured to determine a medical treatment.

[0018] For example, in Example 1 or 2, the processor may be further configured to determine a medical treatment based on context data.

[0019] 3. The surgical hub according to Example 1 or 2, wherein the processor is further configured to determine display data related to a second surgical operation using a medical device based on user identification information and context data.

[0020] For example, in any of Examples 1 to 3, sending a message instructing the display to prioritize display data associated with a second surgical operation may include determining or generating display data associated with the second surgical operation based on user identification information and context data, and sending a message instructing the display to prioritize the display data.

[0021] 4. The surgical hub according to any one of Examples 1 to 3, wherein the processor is further configured to determine display data related to a user performing a second surgical operation using a medical device.

[0022] For example, in any of Examples 1 to 4, sending a message instructing the display to prioritize display data associated with a second surgical operation may include determining or generating display data associated with the second surgical operation based on the user performing the second surgical operation, and sending a message instructing the display to prioritize the display data.

[0023] For example, in any of Examples 1 to 4, the processor may be configured to determine or generate display data, and the display data is related to a user performing a second surgical operation using a medical device.

[0024] For example, in any of Examples 1 to 4, the user and / or user identification information may be a nurse, instrument nurse, medical technician, doctor, surgeon, surgical assistant, and / or medical expert.

[0025] For example, in any of Examples 1 to 4, the display data may include steps for the use of surgical instruments, device settings, device status, device commands for use, operation parameters, warnings or error messages, identification of errors, commands to resolve errors, one or more commands to assist the user when performing surgical operations to correct errors, indication of detected abnormalities, preoperative imaging, intraoperative imaging, instrument data, treatment commands, indication of the predicted result when the user executes the intended action, change in the flow of body fluid, blood flow in blood vessels, such as low blood flow or high blood flow, indication of which blood vessels supply blood to the tumor, tumor marking, tumor margin, resection margin, resection or tumor margin adjusted during the surgical procedure, real-time Doppler monitoring, preoperative MRI data, indocyanine green (ICG) fluorescence imaging data, identification information of the user operating the surgical tool, real-time surgical data received from another connected system, proximity alerts when a surgical instrument, such as at least one surgical instrument (or its distal tip), moves within a specific range of important structures, cut portions of blood vessels to represent the blood supply affected by the cutting path, a part of an anatomical structure that can be made non-viable by the cutting path due to lack of blood or air, the cutting path proposed by the hub or processor, indication of expected problems, visualization of the occluded part of the surgical site, landmarks displayed in the image of the surgical site, and / or anatomical identification information generated based on preoperative images.

[0026] 5. The surgical hub according to any one of Examples 1 to 4, wherein a second surgical operation is performed after completion of a first surgical operation of a medical treatment.

[0027] For example, in any of Examples 1 to 5, the second surgical operation may be the next task in a surgical procedure following the first task. Determining the second surgical operation using a medical instrument based on the first surgical operation and contextual data may include determining the second surgical operation using a medical instrument based on contextual data, and / or that the second task is the next step in a surgical procedure following the first task.

[0028] 6. A surgical hub according to any one of Examples 1 to 5, wherein the message is a first message, and the processor is further configured to send a second message to a medical instrument to instruct the medical instrument to be configured according to a second surgical operation.

[0029] 7. A surgical hub according to any one of Examples 1 to 6, wherein the context data includes one or more of the following: data received from a medical device, status of the medical device, status of a subsystem of the medical device, status of a component of the medical device, status of the motor of the medical device, end effector orientation, reload status, configuration of the medical device, and operational information.

[0030] For example, in any of Examples 1 to 34, the context data, first context data, and / or second context data are data received from the medical device, the status of the medical device, the status of the medical device subsystem, the status of the medical device components, the status of the medical device motor, the orientation of the end effector, the reload status, the configuration of the medical device, operating information, an indication that the user is controlling the medical device, an indication that the medical device is in a specific location, an indication that the user is in a specific location, an indication that the user is beyond a threshold distance from the location, an indication that the user is beyond a threshold distance from the device, an indication of the location type or purpose of the location, an indication that the user is exchanging a medical device or other medical device. This may include at least one of the following: an indication that the device is being operated on, an indication that the location is close to or inside the patient, an indication that the location is a surgical site, an indication that the location is inside a trocar, user feedback, user-adjusted parameters of the medical device, standby time, firing force parameters, clamp compression parameters, an indication that the cartridge is loaded, cartridge status, instructions for enabled controls, instructions for disabled medical device controls, battery power level, pre- and post-operative data, device data, image data, medical device data, biometric data, patent data, EMR data, video data, patient data, medical device parameters, images of the surgical site, and / or videos of the surgical site.

[0031] 8. The surgical hub according to any one of Examples 1 to 7, wherein the processor is further configured to determine that a second surgical operation using a medical instrument is one or more of the following: an important task, a critical task, a dangerous task, or an error correction task.

[0032] For example, in Example 8, important, critical, and / or dangerous tasks include ligating the IMA branch, accessing the plane between the reticular and colon, managing the breeder or principal breeder, and freeing the splenic flexure from the reticular and / or spleen and / or colon. This may include at least one of the following steps: resection of the distal sigmoid colon below the rectosigmoid junction; firing a circular stapler below the segment; performing an anastomosis; or a step that was not initially planned as part of the surgical procedure but became necessary.

[0033] 9. A surgical hub for prioritizing data on a display, wherein the surgical hub is Equipped with a processor, the processor, Based on contextual data, determine the first surgical operation in which medical instruments will be used during the medical procedure, Receiving instrument data from medical instruments associated with the first surgical procedure, Based on the first surgical procedure, instrument data, and medical treatment, determine the second surgical procedure using the medical instrument. A surgical hub is configured to send messages to the display instructing it to prioritize display data associated with a second surgical operation.

[0034] For example, in Example 9, the surgical hub may use medical device context recognition to prioritize data on the display.

[0035] For example, in Example 9, the surgical hub may be configured to connect to a medical instrument and a display.

[0036] For example, in Example 9, determining the first surgical procedure may include determining, based on contextual data, that the first surgical procedure is currently being performed, and the first surgical procedure involves the use of medical instruments during a medical procedure.

[0037] For example, in Example 9, the medical device may be at least one of the following: a laparoscope, an endoscope, a thoracoscope, a laparoscopic surgical instrument, an electrosurgical instrument, an ultrasonic surgical instrument, and / or a surgical stapling device.

[0038] For example, in Embodiment 9, the display may be an instrument display located on a medical instrument such as a medical device, a display in or within an operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, a wrist-worn display, an AR device, and / or a head-mounted display. The display may be configured to display data including display data. Sending a message instructing the display to prioritize display data associated with a second surgical operation may include sending a message instructing the display to prioritize display data associated with a second surgical operation over other display data.

[0039] For example, in Embodiment 9, sending a message instructing the display to prioritize display data associated with a second surgical operation may include making the display data larger, highlighting the display data, changing the color of the display data, moving the display data to a more prominent position, making non-display data semi-transparent, changing the color of non-display data, making non-display data smaller, and / or moving non-display data to a less prominent part of the display.

[0040] For example, in Embodiment 9, sending a message instructing the display to prioritize display data associated with the second surgical operation may include determining or generating display data associated with the second surgical operation and sending a message instructing the display to prioritize the display data.

[0041] For example, in Example 9, the displayed data includes steps for using a surgical instrument, device settings, device status, device commands for use, operating parameters, warnings or error messages, error identification, commands to resolve errors, one or more commands to assist the user in performing surgical work to correct errors, indication of detected abnormalities, preoperative imaging, intraoperative imaging, instrument data, procedure commands, indication of expected results if the user performs the intended action, changes in fluid flow, blood flow in blood vessels, e.g., low or high blood flow, indication of which blood vessels supply blood to the tumor, tumor marking, tumor margins, resection margins, resection or tumor margins adjusted during surgical procedure, real-time Doppler monitoring, and hand This may include preoperative MRI data, indocyanine green (ICG) fluorescence imaging data, user identification information for operating surgical tools, real-time surgical data received from another connected system, proximity alerts when a surgical instrument, such as at least one surgical instrument (or its distal tip), moves within a specific range of a critical structure, severed portions of blood vessels to represent blood supply affected by the cutting route, portions of anatomical structures that could be rendered inviolable by the cutting route due to blood or air deficiency, cutting routes proposed by the hub or processor, indications of anticipated problems, visualization of occluded portions of the surgical site, landmarks displayed on images of the surgical site, and / or anatomical identification information generated based on preoperative images.

[0042] For example, in Embodiment 9, the displayed data may be associated with the second surgical operation by being related to the second surgical operation, being a procedural command for performing the second surgical operation, being patient data and / or instrument data related to the second surgical operation, being an indication that the second task was performed incorrectly or that an error occurred in the second task, being one or more commands for resolving an error in the second task, or being one or more commands for assisting the user when performing a corrective surgical operation.

[0043] For example, in Embodiment 9, the surgical hub may include a context-aware system configured to collect context-aware data and determine information from the context-aware data. For example, the context-aware system may be configured to determine or measure information such as context data, surgical operations such as a first surgical operation and / or a second surgical operation, medical procedures such as medical procedures, and / or user and / or user identification information. The context-aware system may include or be configured to connect at least one piece of hardware to transmit context-aware data to the context-aware system. The hardware may include at least one of a camera, an in-operating sensor, a sensor associated with a user, a sensor associated with or part of a medical instrument, and / or a wearable device.

[0044] For example, in Example 9, the processor may be further configured to determine a medical procedure based on contextual data.

[0045] 10. The surgical hub as described in Example 9, wherein the instrument data includes one or more of the following: user feedback, user-adjusted parameters for the medical device, standby time, force-to-fire parameter (FTF), clamp compression parameter, indicator of cartridge loading, cartridge status, indicator of activated control, indicator of deactivated medical device control, battery power level, and medical device status.

[0046] For example, in Embodiment 9 or 10, the instrument data may include at least one of the following: user feedback, user-adjusted medical instrument parameters, standby time, firing force parameter (FTF), clamp compression parameter, indication that a cartridge is loaded, cartridge status, indication of enabled control, indication of disabled medical instrument control, battery power level, medical instrument status, status of medical instrument subsystems, status of medical instrument components, status of medical instrument motor, end effector orientation, reload status, medical instrument configuration, operating information, indication that the user is controlling the medical instrument, indication that the medical instrument is in a specific location, indication that the user is in a specific location, indication that the location is close to or inside the patient, indication that the location is a surgical site, indication that the location is inside the trocar, image data, video data, images of the surgical site, and / or videos of the surgical site.

[0047] 11. The surgical hub according to Example 9 or 10, further configured to determine errors by analyzing instrument data from a medical instrument using contextual data.

[0048] For example, in any of Examples 9 to 11, the processor uses contextual data to analyze instrument data from medical devices and determine if the patient has a critical status, biometric, The system may be further configured to determine an error as a result of at least one of the following: data being out of range; one or more devices being within close range of a potential impact or having been impacted; a bleeding event occurring; detection of an anomaly related to a surgical procedure; receiving surgical data that is outside the expected range; receiving system parameters that are outside the desired system parameter range; and / or detecting a step for use that is out of sequence.

[0049] 12. The surgical hub described in Example 11, in which the displayed data indicates an error.

[0050] 13. The surgical hub according to Embodiment 11 or 12, wherein the processor is further configured to determine one or more instructions for resolving an error, and the display data includes one or more instructions for resolving the error.

[0051] 14. Context data is the first context data, and the processor, Receiving the second context data, The surgical hub according to Example 9 or 10 is further configured to determine an error that occurred during a medical procedure based on second contextual data.

[0052] 15. The surgical hub according to any one of Examples 9 to 14, wherein the second surgical operation is a corrective surgical operation to correct an error that occurred during a medical procedure, and the processor is further configured to determine one or more instructions to assist the user in performing the corrective surgical operation, and the display data includes one or more instructions to assist the user in performing the corrective surgical operation.

[0053] For example, in either of Examples 14 and 15, determining that an error occurred during a medical procedure based on second context data may include determining that an error occurred during a first medical task based on second context data. For example, in Example 15, determining a second surgical operation using a medical instrument may include determining a corrective surgical operation to correct an error that occurred during the first surgical operation and / or medical procedure, based on the error, instrument data, and medical procedure.

[0054] For example, in any of Examples 11 to 15, the processor may determine one or more instructions for resolving the error by determining how the first and / or second surgical operations may have affected the surgical procedure, then determining what set of actions may be used to correct the error, determining that the medical device has experienced an error and determining instructions on how to correct the error on the medical device, providing instructions to the user on how to correct the medical device error, and providing instructions to the healthcare provider on how to repair the components of the medical device so that the medical device can function properly.

[0055] 16. A surgical hub for prioritizing data on a display, Equipped with a processor, the processor, To determine the first surgical operation in which medical instruments will be used during the medical procedure, Based on contextual data, determine if an error occurred during a medical procedure, Based on errors, contextual data, and medical procedures, determine the second surgical operation using medical instruments. Sending a first message to the first display instructs it to display an error indicator, A surgical hub is configured to send a second message instructing a second display to display data associated with a second surgical operation.

[0056] For example, in Example 16, the surgical hub may use medical device status recognition to determine that an error has occurred.

[0057] For example, in Example 16, the surgical hub may be configured to connect to a medical instrument, a first display, and / or a second display.

[0058] For example, in Example 16, determining a first surgical procedure may include determining, based on contextual data, that a first surgical procedure is currently being performed, and that the first surgical procedure involves the use of medical instruments during a medical procedure. The processor may be further configured to determine a medical procedure based on contextual data.

[0059] For example, in Example 16, the medical device may be at least one of the following: a laparoscope, an endoscope, a thoracoscope, a laparoscopic surgical instrument, an electrosurgical instrument, an ultrasonic surgical instrument, and / or a surgical stapling device.

[0060] For example, in Example 16, the first and / or second display may be an instrument display placed on a medical device such as a medical instrument, a display in or within an operating room, a personal display, a television, a computer screen, a personal computer, a tablet, a smartphone, a wrist-worn display, an augmented reality (AR) device, and / or a head-mounted display. The display may be configured to display data including display data.

[0061] For example, in any of the 16 examples, the displayed data includes steps for using a surgical instrument, device settings, device status, device commands for use, operating parameters, warnings or error messages, error identification, commands to resolve errors, one or more commands to assist the user in performing surgical work to correct errors, indication of detected abnormalities, preoperative imaging, intraoperative imaging, instrument data, procedure commands, indication of expected results if the user performs the intended action, changes in fluid flow, intravascular blood flow, e.g., low or high blood flow, indication of which blood vessels supply blood to the tumor, tumor marking, tumor margins, resection margins, resection or tumor margins adjusted during surgical procedure, and real-time Doppler imaging. This may include visual information, preoperative MRI data, indocyanine green (ICG) fluorescence imaging data, user identification information for operating surgical tools, real-time surgical data received from another connected system, proximity alerts when a surgical instrument, such as at least one surgical instrument (or its distal tip), moves within a specific range of a critical structure, severed portions of blood vessels to represent blood supply affected by the cutting route, portions of anatomical structures that could be rendered inviolable by the cutting route due to blood or air deficiency, cutting routes proposed by the hub or processor, indications of anticipated problems, visualization of occluded portions of the surgical site, landmarks displayed on images of the surgical site, and / or anatomical identification information generated based on preoperative images.

[0062] For example, in Example 16, the displayed data may be associated with the second surgical operation by being related to the second surgical operation, being a procedural command for performing the second surgical operation, being patient data and / or instrument data related to the second surgical operation, being an indication that the second task was performed incorrectly or that an error occurred in the second task, being one or more commands for resolving an error in the second task, or being one or more commands for assisting the user when performing a corrective surgical operation.

[0063] For example, in Embodiment 16, the surgical hub may include a context-aware system configured to collect context-aware data and determine information from the context-aware data. For example, the context-aware system may be configured to determine or measure information such as context data, surgical operations such as a first surgical operation and / or a second surgical operation, medical procedures such as medical procedures, and the occurrence of errors. The context-aware system may include or be configured to connect at least one piece of hardware to transmit context-aware data to the context-aware system. The hardware may include at least one of a camera, an in-operating room sensor, a sensor associated with a user, a sensor associated with or part of a medical instrument, and / or a wearable device. For example, in Embodiment 16, the processor may be configured to determine an error as a result of at least one of the following: the patient has a critical status, biometric data is out of range and one or more devices are within close range that could cause an impact or have been impacted, a bleeding event has occurred, detection of an anomaly related to a surgical procedure, receiving surgical data that is out of expected range, receiving system parameters that are out of desired range, and / or detection of a step for use that is out of sequence.

[0064] 17. The surgical hub according to Example 16, wherein the first display is a primary display and the second display is a secondary display associated with a medical device.

[0065] For example, in Example 17, the medical device includes a secondary display.

[0066] 18. The surgical hub according to Example 16 or 17, wherein the processor is further configured to determine a solution to an error, and the displayed data includes the solution to the error.

[0067] For example, in any of Examples 16 to 18, the processor may be configured to determine one or more instructions for resolving an error, and the display data includes one or more instructions for resolving the error.

[0068] 19. The surgical hub according to any one of Examples 16 to 18, wherein the second surgical operation is a corrective surgical operation for correcting an error, and the processor is further configured to determine one or more instructions to assist the user in performing the corrective surgical operation, and the display data includes one or more instructions to assist the user in performing the corrective surgical operation.

[0069] For example, in any of Examples 16 to 19, determining that an error occurred during a medical procedure based on second context data may include determining that an error occurred during a first medical task based on context data.

[0070] 20. The surgical hub according to any one of Examples 16 to 19, wherein the second message further instructs the display to show instructions to the user to assist the user in resolving the error.

[0071] For example, in any of Examples 16 to 20, the displayed data may be instructions to the user to assist the user in resolving errors.

[0072] 21. The surgical hub according to any one of Examples 16 to 20, wherein the second message further instructs the display to show the user a correction command, the correction command includes one or more of the following: a medical device cleaning command, a medical device reload command, and a medical device repair command.

[0073] For example, in any of Examples 19 to 21, the processor may determine one or more instructions to assist the user in performing a corrective surgical operation by determining how the first and / or second surgical operation may have affected the surgical procedure, then determining what action policies may be used to correct the error, determining that the medical device has experienced an error and determining instructions on how to correct the error on the medical device, providing the user with instructions on how to correct the medical device error, and providing the healthcare provider with instructions on how to repair the components of the medical device so that the medical device can function properly.

[0074] 22. A medical device for prioritizing data on a display, wherein the medical device is a display and A processor, and the processor, Determining context data, Based on contextual data, determine the surgical procedure in which medical instruments are used during the medical treatment, To determine display data that is associated with surgical procedures and related to users performing surgical procedures using medical instruments, A medical device configured to send messages to a display instructing it to prioritize display data associated with surgical procedures.

[0075] For example, in Example 22, the medical device may use situational awareness and the medical device to prioritize data on the display.

[0076] For example, in Example 22, determining a surgical procedure may include determining, based on contextual data, that a surgical procedure is about to be performed, and that the surgical procedure involves the use of medical instruments during the medical procedure.

[0077] For example, in Example 22, the medical device may be at least one of the following: a laparoscope, an endoscope, a thoracoscope, a laparoscopic surgical instrument, an electrosurgical instrument, an ultrasonic surgical instrument, and / or a surgical stapler.

[0078] For example, in Embodiment 22, the display may be configured to display data including display data. Sending a message instructing the display to prioritize display data associated with a surgical operation may include sending a message instructing the display to prioritize display data associated with a surgical operation over other display data.

[0079] For example, in Example 22, the user and / or user identification information may be a nurse, scrub nurse, medical technician, physician, surgeon, surgical assistant, and / or medical professional.

[0080] For example, in Example 22, the displayed data includes steps for using a surgical instrument, device settings, device status, device commands for use, operating parameters, warnings or error messages, error identification, commands to resolve errors, one or more commands to assist the user in performing surgical work to correct errors, indication of detected abnormalities, preoperative imaging, intraoperative imaging, instrument data, procedure commands, indication of expected results if the user performs the intended action, changes in fluid flow, blood flow in blood vessels, e.g., low or high blood flow, indication of which blood vessels supply blood to the tumor, tumor marking, tumor margins, resection margins, resection or tumor margins adjusted during the surgical procedure, and real-time data. This may include time-Doppler monitored preoperative MRI data, indocyanine green (ICG) fluorescence imaging data, user identification information for operating surgical tools, real-time surgical data received from another connected system, proximity alerts when surgical instruments (or their distal tips) move within a specific range of critical structures, severed portions of blood vessels to represent blood supply affected by the severance route, portions of anatomical structures that may be rendered inviolable by the severance route due to blood or air deficiency, the severance route, indications of anticipated problems, visualization of occluded portions of the surgical site, landmarks displayed on images of the surgical site, and / or anatomical identification information generated based on preoperative images.

[0081] For example, in Example 22, sending a message instructing the display to prioritize display data associated with surgical work may include making the display data larger, highlighting the display data, changing the color of the display data, moving the display data to a more prominent position, making non-display data semi-transparent, changing the color of non-display data, making non-display data smaller, and / or moving non-display data to a less prominent part of the display.

[0082] For example, in Example 22, the displayed data may be associated with a surgical operation by being related to the surgical operation, being a procedural command for performing the surgical operation, being patient data and / or instrument data related to the surgical operation, being an indication that the task was performed incorrectly or that an error occurred in the task, being one or more commands for resolving an error in the task, or being one or more commands for assisting the user when performing a corrective surgical operation.

[0083] For example, in Example 22, the medical device may include a situational awareness system configured to collect situational awareness data and determine information from the situational awareness data, or may be configured to connect to a situational awareness system. For example, the situational awareness system may be configured to determine or measure information such as contextual data, surgical procedures, medical procedures, and / or user and / or user identification information. The situational awareness system may include or be configured to connect to at least one piece of hardware for transmitting situational awareness data to the situational awareness system. The hardware may include at least one of a camera, an in-operating room sensor, a sensor associated with the user, a sensor associated with or part of the medical device, and / or a wearable device.

[0084] 23. The medical device according to Example 22, wherein the surgical operation is one or more of the following tasks: reloading the medical device, preparing the medical device, cleaning the medical device, testing the medical device, handing the medical device to another user, repairing the medical device, determining a medical device error, and using the medical device to treat a patient.

[0085] For example, in any of Examples 1 to 34, the surgical procedure may include at least one of the following tasks: reloading a medical instrument, preparing a medical instrument, cleaning a medical instrument, testing a medical instrument, handing a medical instrument to another user, repairing a medical instrument, determining a medical instrument error, and / or performing a procedure on a patient using the medical instrument.

[0086] 24. A medical device according to Example 22 or 23, further comprising determining a medical procedure.

[0087] For example, in Example 24, the processor may be further configured to determine a medical procedure based on contextual data.

[0088] 25. A medical instrument according to any one of Examples 22 to 24, wherein the surgical operation is a first surgical operation, and the processor is further configured to determine a second surgical operation in which the medical instrument is used.

[0089] 26. A medical device according to any one of Examples 22 to 24, wherein context data is first context data, surgical operation is first surgical operation, and the processor is further configured to determine a second surgical operation in which a medical device is used during a medical procedure, based on second context data.

[0090] 27. A medical device according to any one of Examples 22 to 26, wherein the message is a first message, the display data is first display data, and the processor is further configured to determine second display data associated with a second surgical operation and relating to a user performing a second surgical operation using a medical device, based on user identification information and second context data.

[0091] 28. The medical device according to Embodiment 27, further configured to send a second message instructing the display to prioritize the second display data over the first display data.

[0092] 29. A medical device according to any one of Examples 22 to 24, wherein context data is first context data, surgical operation is first surgical operation, and the processor is further configured to determine a second surgical operation in which a medical device is used during a medical procedure, based on the first surgical operation, second context data, and medical procedure.

[0093] In Examples 25-34, all comments relating to Example 22 concerning surgical operations, context data, messages, display data, and prioritization are applied mutatis mutandis to the second surgical operation, second context data, second message, second display data, and further prioritization, respectively.

[0094] 30. A medical device according to any one of Examples 22 to 29, wherein the processor is further configured to determine that a surgical operation using the medical device is one or more of the following: an important task, a critical task, a dangerous task, or an error correction task.

[0095] For example, in Example 22, important, critical, and / or dangerous tasks may include ligating the IMA branch, accessing the plane between the retina and the colon, managing the breeder or major breeder, freeing the splenic flexure from the retina and / or spleen and / or colon, resecting the distal sigmoid colon below the rectosigmoid junction, firing a circular stapler below the segment, performing an anastomosis, and at least one of the steps that were not initially planned as part of the surgical procedure but became necessary.

[0096] 31. A medical device according to any one of Examples 22 to 30, wherein the displayed data includes one or more of the following: parameters for the medical device adjusted by the user, standby time, firing force parameter (FTF), clamp compression parameter, indicator that a cartridge is loaded, cartridge status, indicator of activated control, indicator of deactivated medical device control, battery power level, and status of the medical device.

[0097] 32. A medical device according to any one of Examples 22 to 31, wherein the displayed data includes one or more instructions that instruct or assist the user in performing a surgical procedure using the medical device.

[0098] 33. Context data is the first context data, display data is the first display data, message is the first message, and the processor is Based on the second context data, it is determined that an error occurred during the medical procedure, Based on the error, determine a second surgical procedure using a medical instrument, and determine second display data associated with the second surgical procedure and related to the user performing the second surgical procedure using the medical instrument. The medical device according to any one of Examples 22 to 32, further configured to send a second message instructing the display to again prioritize the second display data over the first display data.

[0099] For example, in Embodiment 33, the processor may be configured to determine an error as a result of at least one of the following: the patient has a critical status; biometric data is out of range; one or more devices are within close proximity that could cause an impact, or have been impacted; a bleeding event has occurred; an anomaly related to a surgical procedure has been detected; surgical data is out of expected range; system parameters are out of desired range; and / or a step for use is out of sequence.

[0100] 34. The medical device according to Embodiment 33, wherein the second surgical operation is a corrective surgical operation for correcting an error, and the processor is further configured to determine one or more instructions to assist the user in performing the corrective surgical operation, and the second display data includes one or more instructions to assist the user in performing the corrective surgical operation.

[0101] For example, in any of Examples 22 to 34, determining that an error occurred during a medical procedure based on second context data may include determining that an error occurred during a surgical operation based on second context data. For example, in Example 34, determining a second surgical operation using a medical instrument may include determining a corrective surgical operation to correct an error that occurred during the first surgical operation and / or medical procedure.

[0102] For example, in any of Examples 22 to 34, the processor may determine one or more instructions to assist the user in performing a second surgical operation, a corrective surgical operation, by determining how the first and / or second surgical operation may have affected the surgical procedure, then determining what action policies may be used to correct the error, determining that the medical device has experienced an error and determining instructions on how to correct the error on the medical device, providing the user with instructions on how to correct the medical device error, and providing the healthcare provider with instructions on how to repair the components of the medical device so that the medical device can function properly. [Brief explanation of the drawing]

[0103] [Figure 1] This is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 2] A surgical system used to perform surgical procedures in an operating room, according to at least one aspect of this disclosure. [Figure 3] A visualization system, a robotic system, and a surgical hub paired with an intelligent instrument, according to at least one aspect of the present disclosure. [Figure 4] The present disclosure, in at least one aspect, describes a surgical data network comprising a modular communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] This disclosure shows a computer-implemented interactive surgical system according to at least one aspect of this disclosure. [Figure 6] The present disclosure shows a surgical hub comprising a plurality of modules connected to a modular control tower, according to at least one aspect of this disclosure. [Figure 7] A logic diagram of a control system for a surgical instrument or tool according to at least one aspect of this disclosure is shown. [Figure 8] This disclosure describes a surgical instrument or tool comprising multiple motors that can be activated to perform various functions, according to at least one aspect of this disclosure. [Figure 9] This is a diagram of a situational awareness surgical system according to at least one aspect of the present disclosure. [Figure 10] An exemplary surgical procedure and inference timeline that a surgical hub can create from data detected at each step of a surgical procedure is shown according to at least one aspect of this disclosure. [Figure 11] This is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 12] A block diagram showing a functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 13] A block diagram of a computer-implemented interactive surgical system configured to adaptively generate control program updates for modular devices, according to at least one aspect of this disclosure, is shown. [Figure 14] The present disclosure shows a surgical system comprising a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter, according to at least one aspect of the present disclosure. [Figure 15A] This section describes an exemplary flow for determining the operating mode and for operating in the determined mode. [Figure 15B] This shows an exemplary flow for changing the operating mode. [Figure 16] This shows the primary display of the surgical hub. [Figure 17] An example of a primary display for a surgical hub is shown. [Figure 18] The diagram shows four wide-angle view images of the surgical site at four separate time points during the procedure. [Figure 19] This shows an example of an augmented video image of a preoperative video image, enhanced with data that identifies the displayed elements. [Figure 20] This diagram shows an exemplary flowchart of the process for displaying one or more images. [Figure 21] A schematic diagram of a beam source and a combined beam detector system used as an in-operating device control mechanism according to at least one aspect of this disclosure is shown. [Figure 22A] Various types of sterile field control and data entry consoles are shown according to at least one aspect of this disclosure. [Figure 22B] Various types of sterile field control and data entry consoles are shown according to at least one aspect of this disclosure. [Figure 22C] Various types of sterile field control and data entry consoles are shown according to at least one aspect of this disclosure. [Figure 22D] Various types of sterile field control and data entry consoles are shown according to at least one aspect of this disclosure. [Figure 22E] Various types of sterile field control and data entry consoles are shown according to at least one aspect of this disclosure. [Figure 22A] This shows a single-zone sterile field control and data entry console. [Figure 22B] This shows the multi-zone sterile field control and data entry console. [Figure 22C] This shows a tethered sterile field control and data entry console. [Figure 22D] This shows a battery-powered sterile field control and data entry console. [Figure 22E] This shows a battery-powered sterile field control and data entry console. [Figure 23A] A sterile field console in use in a sterile field during a surgical procedure is shown according to at least one aspect of this disclosure. The sterile field console is shown positioned in a sterile field near two surgeons performing surgery. [Figure 23B] The image shows a sterile field console in use in a sterile field during a surgical procedure, according to at least one aspect of the present disclosure. It shows one of the surgeons tapping the touchscreen of the sterile field console. [Figure 24] At least one aspect of this disclosure describes a standard technique for estimating vascular pathways and depths, as well as instrument trajectories. [Figure 25A] The present disclosure shows multiple real-time views of images of virtual anatomical details for incision, according to at least one aspect of this disclosure. The images are perspective views of the virtual anatomical details. [Figure 25B] The present disclosure shows multiple real-time views of images of a virtual anatomical detail for incision, according to at least one aspect of this disclosure. These are side views of the virtual anatomical detail. [Figure 25C] The present disclosure shows multiple real-time views of images of virtual anatomical details for incision, according to at least one aspect of this disclosure. The images are perspective views of the virtual anatomical details. [Figure 25D] The present disclosure shows multiple real-time views of images of a virtual anatomical detail for incision, according to at least one aspect of this disclosure. These are side views of the virtual anatomical detail. [Figure 26A] This disclosure shows a touchscreen display that may be used in a sterile field according to an aspect of this disclosure. It also shows an image of a surgical site displayed on the touchscreen display in portrait mode. [Figure 26B] An aspect of this disclosure shows a touchscreen display that may be used in a sterile field. The touchscreen display is shown rotated in landscape mode, and the surgeon uses their index finger to scroll through images in the direction of the arrow. [Figure 26C] This exhibit shows a touchscreen display that may be used in a sterile field according to an aspect of the present disclosure. It shows a surgeon pinching and zooming in on an image in the direction of the arrow using their index finger and thumb. [Figure 26D] This exhibit shows a touchscreen display that may be used in a sterile field according to an aspect of the present disclosure. It shows a surgeon pinching and zooming out of an image in the direction of the arrow using their index finger and thumb. [Figure 26E]A touchscreen display that may be used in a sterile field according to an aspect of this disclosure is shown. The touchscreen display is shown rotated in two directions indicated by arrows so that the surgeon can view the image in different orientations. [Figure 27] This is a logical flowchart of a process showing a control program or logical configuration for communicating from inside a sterile field to an apparatus located outside a sterile field, according to at least one aspect of the present disclosure. [Figure 28] A second information layer overlays a first information layer, according to at least one aspect of this disclosure. [Figure 29] The image shows a perspective view of a surgeon using a surgical instrument, including a handle assembly housing and a wireless circuit board, during a surgical procedure, with the surgeon wearing safety goggles, according to at least one aspect of the present disclosure. [Figure 30] This disclosure provides a method, according to at least one aspect thereof, for identifying surgical data associated with a failure event and communicating the identified surgical data to a cloud-based system on a priority basis. [Figure 31] An ultrasonic ping delivery of an operating room wall for determining the distance between a surgical hub and the operating room wall is shown according to at least one aspect of the present disclosure. [Figure 32] This is a logical flowchart of a process illustrating a control program or logical configuration for surgical hub pairing with surgical devices of a surgical system located within the boundaries of an operating room, according to at least one aspect of the present disclosure. [Figure 33] This is a logic flow diagram of a process illustrating a control program or logical configuration for selectively forming and severing connections between devices in a surgical system, according to at least one aspect of the present disclosure. [Figure 34] This is a logical flowchart of a process illustrating a control program or logical configuration for selectively re-evaluating the boundaries of an operating room after detecting a new device, according to at least one aspect of the present disclosure. [Figure 35]This is a logical flowchart illustrating a process for illustrating a control program or logical configuration for selectively re-evaluating the boundaries of an operating room after a paired device has been disconnected, according to at least one aspect of the present disclosure. [Figure 36] This is a logical flowchart illustrating a process for illustrating a control program or logical configuration for a surgical hub to re-evaluate the boundaries of an operating room after detecting a change in the position of the surgical hub, according to at least one aspect of the present disclosure. [Figure 37] This is a logical flow diagram of a process illustrating a control program or logical configuration for selectively forming connections between devices of a surgical system, according to at least one aspect of the present disclosure. [Figure 38] This is a logic flow diagram of a process illustrating a control program or logical configuration for selectively forming and severing connections between devices in a surgical system, according to at least one aspect of the present disclosure. [Figure 39] A surgical hub for pairing a first device and a second device of a surgical system in an operating room is shown according to at least one aspect of the present disclosure. [Figure 40] A surgical hub is shown according to at least one aspect of the present disclosure, which unpairs a first device and a second device of a surgical system in an operating room and pairs the first device with a third device in the operating room. [Figure 41] A logic flow diagram of a process illustrating a control program or logical configuration for forming and severing connections between devices of a surgical system in an operating room during a surgical procedure, based on the progress of the steps of the surgical procedure, according to at least one aspect of the present disclosure. [Figure 42] This is a logical flowchart illustrating a control program or logical configuration for overlaying information derived from one or more static frames of a live stream of a remote surgical site onto the live stream, according to at least one aspect of the present disclosure. [Figure 43] This is a logical flow diagram of a process illustrating a control program or logical configuration for distinguishing surgical steps of a surgical procedure, according to at least one aspect of the present disclosure. [Figure 44]This is a logical flow diagram of a process illustrating a control program or logical configuration for distinguishing surgical steps of a surgical procedure, according to at least one aspect of the present disclosure. [Figure 45] This is a logical flow diagram of a process illustrating a control program or logical configuration for identifying a staple cartridge from information derived from one or more static frames of staples deployed within an organization, according to at least one aspect of the present disclosure. [Figure 46] A partial diagram of a surgical system in an operating room according to at least one aspect of the present disclosure, the surgical system including a surgical hub having an imaging module that communicates with an imaging device at a remote surgical site. [Figure 47] This disclosure provides at least one aspect of a partially artificial schedule of a surgical procedure performed in an operating room via a surgical system. [Figure 48] An interaction between two surgical hubs in different operating rooms ("OR1" and "OR3") is shown according to at least one aspect of this disclosure. [Figure 49] A secondary display in an operating room ("OR3") showing the surgical site for a colorectal procedure, according to at least one aspect of the present disclosure. [Figure 50] A personal interface or tablet within OR1 that displays the surgical site of OR3, according to at least one aspect of this disclosure. [Figure 51] An enlarged view of the surgical site of OR3 as displayed on the primary display of OR1, according to at least one aspect of this disclosure, is shown. [Figure 52] A personal interface or tablet displaying a layout of OR1, showing an available display, according to at least one aspect of this disclosure. [Figure 53] In at least one aspect of this disclosure, a recommendation for the transverse incision location of the surgical site of OR3, made by the surgeon in OR1, is shown via a personal interface or tablet in OR1. [Figure 54A]A logical flow diagram of a process for controlling a modular device according to contextual information derived from received data, according to at least one aspect of the present disclosure. [Figure 54B] This is a logical flowchart of a process for controlling a second modular device according to contextual information derived from pre- and post-operative data received from a first modular device, according to at least one aspect of the present disclosure. [Figure 54C] This is a logical flowchart of a process for controlling a second modular device according to contextual information derived from pre- and post-operative data received from a first modular device and a second modular device, according to at least one aspect of the present disclosure. [Figure 54D] This is a logical flowchart of a process for controlling a third modular device according to contextual information derived from pre- and post-operative data received from a first modular device and a second modular device, according to at least one aspect of the present disclosure. [Figure 55] This is a logical flow diagram for tracking data associated with an operating room event, according to at least one aspect of the present disclosure. [Figure 56] This is a schematic diagram of a robotic surgical system during a surgical procedure, including multiple hubs and interactive secondary displays, according to at least one aspect of the present disclosure. [Figure 57] This is a detail view of the interactive secondary display of Figure 57, according to at least one aspect of the present disclosure. [Figure 58] This diagram illustrates the pairing of a privately owned wireless device with a surgical hub according to at least one aspect of the present disclosure. [Figure 59] This is a diagram of an exemplary operating room (OR) setup according to at least one aspect of the present disclosure. [Figure 60] This is a logical flowchart of a process for visually evaluating surgical staff, according to at least one aspect of the present disclosure. [Figure 61]This figure shows a series of models of surgical staff members during a surgical procedure, according to at least one aspect of the present disclosure. [Figure 62] A graph showing the measured posture of the surgical staff member shown in Figure 61 over time, according to at least one aspect of this disclosure. [Figure 63] This figure shows a surgeon holding a surgical instrument, according to at least one aspect of the present disclosure. [Figure 64] A scatter plot of wrist angles relative to surgical procedure outcomes, according to at least one aspect of the present disclosure. [Figure 65A] This is a logical flow diagram of a process for controlling a surgical device according to at least one aspect of the present disclosure. [Figure 65B] This is a logical flow diagram of a process for generating surgical metadata according to at least one aspect of the present disclosure. [Figure 66] This is a block diagram of a gesture recognition system according to at least one aspect of the present disclosure. [Figure 67] This is a logical flowchart of the process for controlling a display using situational awareness of medical devices. [Figure 68] This figure shows one or more displays that can be controlled using situational awareness of one or more medical instruments during the course of a surgical procedure. [Figure 69] This is a logical flowchart of the process for controlling the display using contextual awareness to prioritize the data shown to the user. [Figure 70] This is a logical flowchart of the process for displaying information on a screen based on the user's visual focus. [Figure 71] The diagram illustrates one or more displays that can display information based on the user's visual focus. [Figure 72] The diagram illustrates one or more displays capable of displaying information based on the visual focus of one or more users. [Figure 73]This is a logical flow diagram of the process for creating the data displayed on the screen. [Figure 74] This is a logical flowchart of the process for controlling displays that may be located outside the sterile field. [Modes for carrying out the invention]

[0104] The applicant of this application also owns the following U.S. patent applications filed concurrently, each of which is incorporated herein by reference in whole: U.S. Patent Application No. 16 / 209,416, filed on December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS"; U.S. Patent Application No. 15 / 940671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER," filed on March 29, 2018 (Agent Reference Number END8502USNP); U.S. Patent Application No. 16 / 182,269, filed on November 6, 2018, entitled "IMAGE CAPTURING THE AREAS OUTSIDE THE ABDOMINAL TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE" (Agent Reference Number: END9018USNP3); U.S. Patent Application No. 16 / 729,747, entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS," filed on December 31, 2019 (Agent Reference Number: END9217USNP1); U.S. Patent Application No. 16 / 729,778, titled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE," filed on December 31, 2019 (Agent Reference Number: END9219USNP1); U.S. Patent Application No. 16 / 729,807, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY," filed on December 31, 2019 (Agent Reference Number: END9228USNP1); U.S. Patent Application No. 15 / 940654, titled "SURGICAL HUB SITUATIONAL AWARENESS," filed on March 29, 2018 (Agent Reference Number END8501USNP); U.S. Patent Application No. 15 / 940,704, filed on March 29, 2018, entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT" (Agent Reference Number END8504USNP); U.S. Patent Application No. 16 / 182,290, filed on November 6, 2018, entitled "SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION" (Agent Reference Number END9018USNP5); • U.S. Patent No. 9011427, titled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued on April 21, 2015; • U.S. Patent No. 9, 123,155, issued on September 1, 2015, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES"; U.S. Patent Application No. 16 / 209,478, filed on December 4, 2018, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE" (Agent Reference Number END9015USNP1); U.S. Patent Application No. 16 / 182,246, entitled "ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES," filed on November 6, 2018 (Agent Reference Number END9016USNP1); U.S. Patent Application No. 16 / 209,385, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," filed on December 4, 2018 (Agent Reference Number END8495USNP); U.S. Patent Application No. 16 / 209,407, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," filed on December 4, 2018 (Agent Reference Number END8497USNP); U.S. Patent Application No. 16 / 182,231, filed on November 6, 2018, entitled "WIRELESS PAIRING OF A SURGICAL DEVICE WITH ANOTHER DEVICE WITHIN A STERILE SURGICAL FIELD BASED ON THE USAGE AND SITUATIONAL AWARENESS OF DEVICES" (Agent Reference Number END9032USNP2); U.S. Patent Application No. 16 / 209,490, entitled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION," filed on December 4, 2018 (Agent Reference Number END9017USNP1); • U.S. Patent Application Publication No. 2014 / 0263552, titled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," published on September 18, 2014; U.S. Patent Application No. 15 / 628175, filed on June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT" (Agent Reference Number END8199USNP); • U.S. Patent Application Publication No. 2009 / 0046146, entitled "SURGICAL COMMUNICATION AND CONTROL SYSTEM," published on February 19, 2009; and • U.S. Patent No. 9,283,054, titled "SURGICAL APPARATUS WITH INDICATOR," issued on March 15, 2016.

[0105] Referring to Figure 1, the computer-implemented interactive surgical system 100 may include one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 which may include a remote server 113 connected to a storage device). Each surgical system 102 may include at least one surgical hub 106 that communicates with the cloud 104 which may include the remote server 113. In one example, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some embodiments, the surgical system 102 may include M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P can be integers of 1 or more.

[0106] In various embodiments, the visualization system 108 may include one or more imaging sensors strategically positioned relative to a sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described in U.S. Patent Application Publication No. 2019-0200844(A1), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0107] As shown in Figure 2, the primary display 119 is positioned in the sterile field so that it is visible to the operator of the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may include a first non-sterile display 107 and a second non-sterile display 109, which face each other at an angle. The visualization system 108, guided by the hub 106, is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 may cause the visualization system 108 to display snapshots of the surgical site, such as those recorded by the imaging device 124, on the non-sterile displays 107 or 109, while maintaining live video of the surgical site on the primary display 119. The snapshots on the non-sterile displays 107 or 109 may, for example, allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0108] In one embodiment, the hub 106 is also configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a primary display 119 in the sterile field, which can be viewed by a sterile operator on the operating table. In one example, the input may take the form of modifications to a snapshot displayed on the non-sterile display 107 or 109, which can be transmitted to the primary display 119 by the hub 106.

[0109] Referring to Figure 2, the surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 may also be configured to coordinate the flow of information from the surgical instrument 112 to the display. For example, the disclosure thereof is incorporated herein by reference in U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY". Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 can be transmitted by the hub 106 to the surgical instrument display 115 in the sterile field, where it can be viewed by the operator of the surgical instrument 112. Examples of surgical instruments suitable for use with surgical system 102 are described, for example, under the heading "Surgical Instrument Hardware" and in U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0110] Figure 2 shows an example of a surgical system 102 used to perform a surgical procedure on a patient lying on an operating table 114 in an operating room 116. A robotic system 110 may be used in the surgical procedure as part of the surgical system 102. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. While the surgeon views the surgical site through the surgeon's console 118, the patient-side cart 120 can operate at least one detachably connected surgical tool 117 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 124, which can be operated by the patient-side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process images of the surgical site, which can then be displayed to the surgeon through the surgeon's console 118.

[0111] Other types of robotic systems can be readily adapted for use with the surgical system 102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed on 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0112] Various examples of cloud-based analytics performed by Cloud 104 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed on 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," which is incorporated herein by reference in its entirety.

[0113] In various embodiments, the imaging device 124 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0114] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0115] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in air from about 380 nm to about 750 nm.

[0116] The invisible spectrum (i.e., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0117] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0118] The imaging device employs multispectral monitoring to distinguish topography and underlying structures. Multispectral images capture image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical work is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding a patient ready for surgery. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and fixation devices within that area.

[0119] Referring here to Figure 3, a hub 106 is shown that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during surgery can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The modular enclosure 136 of the hub provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from the remote surgical site to a suction and irrigation module slidably received within the hub enclosure. In one embodiment, the hub enclosure comprises a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house various generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and second data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a modular enclosure 136 of a hub that enables modular integration of a generator module 140, a smoke exhaust module 126, and a suction / irrigation module 128. The modular enclosure 136 of the hub further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact via the modular enclosure 136 of the hub. The modular enclosure 136 of the hub may be configured to facilitate the insertion of multiple generators and interactive communication between generators docked to the modular enclosure 136 of the hub, so that multiple generators function as a single generator.

[0120] Figure 4 shows a surgical data network 201 comprising a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud-based system (a cloud 204 which may include, for example, a remote server 213 connected to a storage device). In one embodiment, the modular communication hub 203 comprises a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be connected to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional mechanisms that constitute each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0121] Modular devices 1a-1n located in the operating room may be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and operation. Data associated with devices 1a-1n may also be transferred to a local computer system 210 for local data processing and operation. Modular devices 2a-2m located in the same operating room may also be connected to a network switch 209. The network switch 209 may be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to a cloud 204. Data associated with devices 2a-2n may be transferred to a cloud 204 via the network router 211 for data processing and operation. Data associated with devices 2a-2m may also be transferred to a local computer system 210 for local data processing and operation.

[0122] It will be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accommodate multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, the devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module.

[0123] In one embodiment, the surgical data network 201 may include a combination of a network hub, network switches, and network routers connecting devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” can be used as a metaphor for “Internet,” but the term is not limited in that way. Thus, the term “cloud computing” can be used herein to refer to “a type of internet-based computing,” in which various services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 203 and / or computer system 210 located in the surgical field (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 203 and / or computer system 210. The cloud infrastructure may be maintained by a cloud service provider. In this context, a cloud service provider may be an entity that coordinates the use and control of one or more devices 1a-1n / 2a-2m located within the operating room. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located within the operating room. The hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0124] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and increased patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures and to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204, a local computer system 210, or both for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be carried out for tissue-specific sites and conditions. Such data analysis may also involve further prognostic analysis, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0125] The operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of the devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a-1n located within the same operating room network. The network hub 207 can collect data in packet form and transmit them to the router in half-duplex mode. The network hub 207 does not need to store any media access control / Internet Protocol (MAC / Internet Protocol, IP) for transferring any device data. Only one of the devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 does not need to have a routing table or intelligence regarding the destination of the information and broadcasts all network data to each connection and to the remote server 213 (Figure 4) on the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and could cause bottlenecks.

[0126] Operating room devices 2a-2m may be connected to network switch 209 via a wired or wireless channel. Network switch 209 operates within the data link layer of the OSI model. Network switch 209 may also be a multicast device for connecting devices 2a-2m located in the same operating room to a network. Network switch 209 can transmit data in the form of frames to network router 211 and operates in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0127] The network hub 207 and / or network switch 209 may be connected to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates routes for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by any one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, such as different networks located in different operating rooms of the same medical facility or different operating rooms of different medical facilities. The network router 211 can send data in packet form to the cloud 204 and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transfer data.

[0128] In one example, the network hub 207 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0129] In this example, the operating room devices 1a-1n / 2a-2m can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with the modular communication hub 203 via the Bluetooth wireless technology standard to build a personal area network (PAN). The operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via a number of wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G and beyond. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0130] The modular communication hub 203 can function as a central connection point for one or all of the operating room devices 1a-1n / 2a-2m and may handle a data type known as a frame. Frames can carry data generated by the devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, it is amplified and transmitted to the network router 211, which then transfers this data to cloud computing resources using a number of wireless or wired communication standards or protocols as described herein.

[0131] The modular communication hub 203 may be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room equipment 1a~1n / 2a~2m.

[0132] Figure 5 shows a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204 which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to a plurality of operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 connected to a computer system 210.

[0133] As shown in the example in Figure 5, the modular control tower 236 may be connected to an imaging module 238 connected to an endoscope 239, a generator module 240 which can be connected to an energy device 241, a fume exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 which can optionally be connected to a display 237, and a non-contact sensor module 242. The operating room equipment may be connected to cloud computing resources and data storage via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the device / instrument 235 and the visualization system 208 may be connected to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be connected to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with images and superimposed images.

[0134] Figure 6 shows a surgical hub 206 comprising multiple modules connected to a modular control tower 236. The modular control tower 236 may comprise a modular communication hub 203, such as a network connectivity device, and a computer system 210, for example, local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 203, and data associated with the modules may be transferred to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and local displays 217. Communication to the cloud 204 can be done via either a wired communication channel or a wireless communication channel.

[0135] The surgical hub 206 may use a non-contact sensor module 242 to measure the dimensions of the operating room and generate a map of the surgical site using either an ultrasonic or laser non-contact measuring device. The ultrasonic-based non-contact sensor module may scan the operating room by transmitting bursts of ultrasound and receiving echoes as they bounce off the surrounding walls of the operating room, as described in U.S. Patent Application Publication No. 2019-0200844(A1), filed December 4, 2018, which is incorporated herein by whole by reference, and the sensor module may be configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module may scan the operating room by, for example, transmitting laser light pulses, receiving laser light pulses reflected off the outer walls of the operating room, comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0136] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be connected via a system bus to a communication module 247, storage 248, memory 249, non-volatile memory 250, and an input / output interface 251. The system bus may be any of several types of bus structures, including a memory bus or memory controller, peripheral bus or external bus, and / or local bus, using any variety of bus architectures, including but not limited to a 9-bit bus, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other dedicated bus.

[0137] Processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory with a maximum frequency of 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0138] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0139] System memory includes volatile and non-volatile memory. The basic input / output system (BIOS), which contains basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory can include random-access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0140] The computer system 210 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage devices. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disc ROM devices (CD-ROM), compact disc recordable drives (CD-R drives), compact disc rewritable drives (CD-RW drives), or digital versatile disc ROM drives (DVD-ROM). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0141] It should be understood that the computer system 210 may include software that acts as an intermediary between the user and basic computer resources, as described in a preferred operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications can leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0142] The user can input commands or information to the computer system 210 via input devices connected to the I / O interface 251. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to the output device. Output adapters may be provided to indicate that there may be several output devices, such as monitors, displays, speakers, and printers, among others, that may require special adapters. Examples of output adapters include, but are not limited to, video and sound cards that provide means of connection between the output device and the system bus. Note that other devices and / or systems of devices, such as remote computers, may provide both input and output functions.

[0143] The computer system 210 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to the computer system. For brevity, only memory storage devices are shown along with the remote computers. The remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication connection. The network interface may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include Fiber Distributed Data Interfaces (FDDI), Copper Distributed Data Interfaces (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated services digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0144] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5 and 6, and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction, multiple data (SIMD) or multiple-instruction, multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0145] The communication connection section can refer to the hardware / software used to connect a network interface to a bus. For the sake of clarity of the example, the communication connection section is shown as being inside the computer system, but it may also be outside the computer system 210. For illustrative purposes only, hardware / software required for connecting to a network interface include internal and external technologies such as modems including typical telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0146] Figure 7 shows a logic diagram of a control system 470 for a surgical instrument or tool according to one or more embodiments of the present disclosure. The system 470 may include a control circuit. The control circuit may include a microcontroller 461 having a processor 462 and memory 468. For example, one or more of sensors 472, 474, and 476 provide real-time feedback to the processor 462. A motor 482 driven by a motor driver 492 drives an I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 480 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to the processor 462, which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control the firing of the I-beam, the movement of the occluder, the rotation of the shaft, and joint movement. The display 473 can display various operating conditions of the instrument and may include a touchscreen function for data input. The information displayed on the display 473 can be overlaid with images acquired via the endoscopic imaging module.

[0147] In one embodiment, the microcontroller 461 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 461 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels.

[0148] In one embodiment, the microcontroller 461 may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0149] The microcontroller 461 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 461 may include a processor 462 and memory 468. The electric motor 482 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 492 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 480 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0150] The microcontroller 461 may be programmed to provide precise control over the velocity and position of the displacement member and joint motion system. The microcontroller 461 may be configured to calculate the response within its software. The calculated response is compared with the measured response of the actual system to obtain an "observed" response, which can be used to determine the actual feedback. The observed response may be a well-adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and it can detect external influences on the system.

[0151] In some examples, the motor 482 may be controlled by a motor driver 492 and may be used by a surgical instrument or tool launching system. In various forms, the motor 482 may be, for example, a brushed DC-driven motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 482 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 492 may comprise, for example, an H-bridge driver including a field-effect transistor (FET). The motor 482 may be powered by a power supply assembly removably mounted on a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for supplying power to a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be a lithium-ion battery that can be coupled to and detached from the power supply assembly.

[0152] The motor driver 492 may be the A3941, available from Allegro Microsystems, Inc. The A3941 492 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET), particularly designed for inductive loads such as brushed DC motors. The driver 492 may include a proprietary charge pump regulator, which may provide full (>10V) gate drive to battery voltages down to 7V, and may allow the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge may be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be easily substituted for use in the tracking system 480 with an absolute positioning system.

[0153] The tracking system 480 may include a controlled motor drive circuit arrangement comprising a position sensor 472 according to one aspect of the present disclosure. The position sensor 472 for the absolute positioning system may provide a unique position signal corresponding to the position of the displaced member. In some examples, the displaced member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displaced member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displaced member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displaced member may be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element that can be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 472 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include a contact-type displacement sensor or a non-contact-type displacement sensor.A linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0154] The electric motor 482 may include a rotary shaft that operably interfaces with a gear assembly mounted on a displacement member by meshing with a set of drive teeth or a rack. The sensor element may be operably connected to the gear assembly such that one rotation of the position sensor 472 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply can provide power to the absolute positioning system, and an output indicator can display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing with the corresponding drive gear of the gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0155] One rotation of the sensor element associated with the position sensor 472 may correspond to the longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor mechanism may be connected via a gear reduction that results in the position sensor 472 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 472 can complete multiple rotations relative to the full stroke of the displacement member.

[0156] To provide a unique position signal for two or more rotations of the position sensor 472, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches may be fed back to the microcontroller 461, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 472 is provided to the microcontroller 461. The position sensor 472 of the sensor mechanism may include an array of analog rotation sensors such as a magnetic sensor or potentiometer, or an array of analog Hall effect elements, which output a unique combination of position signals or values.

[0157] The position sensor 472 may comprise any number of magnetic sensing elements, such as magnetic sensors, which are classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may involve numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0158] In one embodiment, the position sensor 472 of a tracking system 480 equipped with an absolute positioning system may be equipped with a magnetic rotation absolute positioning system. The position sensor 472 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 472 interfaces with a microcontroller 461 to provide an absolute positioning system. The position sensor 472 may be a low-voltage, low-power component and includes four Hall effect elements in the area of ​​the position sensor 472 that can be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement a simple and efficient algorithm for calculating hyperbolic and trigonometric functions that requires only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information may be transmitted to the microcontroller 461 via a standard serial communication interface such as a serial peripheral interface (SPI). The position sensor 472 may offer 12-bit or 14-bit resolution. The position sensor 472 may also be an AS5055 chip, available in a small QFN 16-pin 4x4x0.85mm package.

[0159] The tracking system 480, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 472, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of a physical system will be by knowing the input, the calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, and inductive resistance.

[0160] The absolute positioning system can provide the absolute position of a displacement member when the device is powered on, without requiring the displacement member to be moved back or forward to a reset (zero or home) position, as may be necessary with conventional rotary encoders that simply count the number of forward or backward strokes taken by the motor 482 to estimate the position of the device actuator, drive bar, knife, etc.

[0161] Sensor 474, for example, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of an end effector, such as the amplitude of strain exerted on the anvil during clamping, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to processor 462. Instead of, or in addition to, sensor 474, sensor 476, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. Sensor 476, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and deform into contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 478 may be used to measure the current drawn in by motor 482. The force required to propel the launching member forward may correspond, for example, to the current drawn in by the motor 482. The measured force can be converted into a digital signal and provided to the processor 462.

[0162] In one embodiment, a strain gauge sensor 474 can be used to measure the force applied to tissue by the end effector. A strain gauge can be connected to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 474, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 474 can measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 462 of the microcontroller 461. A load sensor 476 can measure the force used to operate a knife element, for example, to cut tissue trapped between an anvil and a staple cartridge. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor can also be converted into a digital signal and provided to the processor 462.

[0163] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 474 and 476 respectively, can be used by the microcontroller 461 to characterize the selected position of the launcher and / or the corresponding values ​​of the launcher's velocity. In one example, memory 468 can store techniques, equations, and / or lookup tables that can be used by the microcontroller 461 during evaluation.

[0164] The control system 470 for surgical instruments or tools may also include a wired communication circuit or a wireless communication circuit for communicating with a modular communication hub 203, as shown in Figures 5 and 6.

[0165] Figure 8 shows a surgical instrument or tool equipped with multiple motors that can be activated to perform various functions. In a particular example, the first motor can be activated to perform the first function, the second motor can be activated to perform the second function, the third motor can be activated to perform the third function, the fourth motor can be activated to perform the fourth function, and so on. In a particular example, the multiple motors of the robotic surgical instrument 600 can be activated individually to produce firing, closing, and / or jointing motions in the end effector. The firing, closing, and / or jointing motions can be transmitted to the end effector, for example, via a shaft assembly.

[0166] In certain examples, the surgical instrument system or tool may include a firing motor 602. The firing motor 602 may be operably connected to a firing motor drive assembly 604, which can be configured to transmit the firing motion generated by the motor 602 to an end effector, specifically to displace an I-beam element. In certain examples, the firing motion generated by the motor 602 may, for example, deploy a staple from a staple cartridge into tissue captured by the end effector and / or advance the cutting edge of the I-beam element to cut the captured tissue. The I-beam element can be retracted by reversing the direction of the motor 602.

[0167] In certain examples, the surgical instrument or surgical tool may include a closing motor 603. The closing motor 603 may be operably coupled to a closing motor drive assembly 605, which may be configured to specifically displace a closing tube to close the anvil and transmit the closing motion generated by the motor 603 to the end effector to compress tissue between the anvil and the staple cartridge. The closing motion allows the end effector to transition from an open configuration to an approach configuration, for example, to capture tissue. The end effector may be moved to an open position by reversing the direction of the motor 603.

[0168] In certain examples, a surgical instrument or tool may include, for example, one or more articular motors 606a, 606b. The motors 606a, 606b may be operably connected to corresponding articular motor drive assemblies 608a, 608b, which may be configured to transmit the articular motion generated by the motors 606a, 606b to an end effector. In certain examples, the articular motion may cause, for example, the end effector to articulate relative to the shaft.

[0169] As described herein, a surgical instrument or tool may include multiple motors that can be configured to perform various independent functions. In certain examples, multiple motors of a surgical instrument or tool can be activated individually or separately to perform one or more functions while other motors remain stopped. For example, articulation motors 606a and 606b can be activated to articulate an end effector while firing motor 602 remains stopped. Alternatively, firing motor 602 can be activated to fire multiple staples and / or advance a cutting edge while articulation motor 606 remains stopped. Furthermore, a closure motor 603 may be activated simultaneously with firing motor 602 to advance the closure tube and I-beam element distally, as described in more detail below herein.

[0170] In certain examples, a surgical instrument or tool may include a common control module 610 that can be used with multiple motors of the surgical instrument or tool. In certain examples, the common control module 610 may correspond to one of the multiple motors at a time. For example, the common control module 610 may be individually connectable and disconnectable to multiple motors of the robotic surgical instrument. In certain examples, multiple motors of the surgical instrument or tool may share one or more common control modules, such as the common control module 610. In certain examples, multiple motors of the surgical instrument or tool can engage with the common control module 610 individually and selectively. In certain examples, the common control module 610 can selectively switch between interfacing with one of the multiple motors of the surgical instrument or tool and interfacing with another of the multiple motors of the surgical instrument or tool.

[0171] In at least one example, the common control module 610 can be selectively switched between an operable engagement with the articulation motors 606a, 606b and an operable engagement with either the firing motor 602 or the closing motor 603. In at least one example, as shown in Figure 8, the switch 614 can move or transition between multiple positions and / or states. For example, in the first position 616, the switch 614 may electrically connect the common control module 610 to the firing motor 602; in the second position 617, the switch 614 may electrically connect the common control module 610 to the closing motor 603; in the third position 618a, for example, the switch 614 may electrically connect the common control module 610 to the first articulation motor 606a; and in the fourth position 618b, the switch 614 may electrically connect the common control module 610 to the second articulation motor 606b. In certain examples, a separate common control module 610 may also be electrically connected to the launch motor 602, the closing motor 603, and the joint motion motors 606a, 606b. In certain examples, the switch 614 may be a mechanical switch, an electromechanical switch, a solid switch, or any preferred switching mechanism.

[0172] Each of the motors 602, 603, 606a, and 606b may be equipped with a torque sensor for measuring the output torque on the motor shaft. The force on the end effector may be sensed in any conventional manner, such as by force sensors on the outside of the jaws or by torque sensors on the motors that actuate the jaws.

[0173] In various examples, as shown in Figure 8, the common control module 610 may include a motor driver 626 which may comprise one or more H-bridge FETs. The motor driver 626 may modulate the power transmitted from the power supply 628 to the motor connected to the common control module 610 based, for example, on input from a microcontroller 620 ("controller"). In certain examples, as described herein, the microcontroller 620 can be used, for example, to determine the current drawn by the motor while the motor is connected to the common control module 610.

[0174] In certain examples, the microcontroller 620 may include a microprocessor 622 ("processor") and one or more non-temporary computer-readable media or memory units 624 ("memory"). In certain examples, the memory 624 may store various program instructions, which, when executed, can cause the processor 622 to perform some of the functions and / or calculations described herein. In certain examples, one or more of the memory units 624 may be linked to the processor 622, for example.

[0175] In certain examples, power supply 628 may be used to supply power to, for example, a microcontroller 620. In certain examples, power supply 628 may include a battery (or "battery pack" or "power pack") such as a lithium-ion battery. In certain examples, the battery pack may be configured to be removably attached to a handle in order to supply power to a surgical instrument 600. Several battery cells connected in series may be used as power supply 628. In certain examples, power supply 628 may be, for example, replaceable and / or rechargeable.

[0176] In various examples, the processor 622 can control the motor driver 626 to control the position, direction of rotation, and / or speed of a motor connected to a common control module 610. In certain examples, the processor 622 can signal the motor driver 626 to stop and / or disable a motor connected to the common control module 610. The term “processor,” as used herein, should be understood to include any suitable microprocessor, microcontroller, or other basic computing device that integrates the functions of a computer’s central processing unit (CPU) on one or up to several integrated circuits. A processor may be a multipurpose programmable device that receives digital data as input, processes that data according to instructions stored in memory, and provides the results as output. It may have internal memory and therefore may be an example of sequential digital logic. A processor may operate with numbers and symbols represented in binary.

[0177] The processor 622 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In a particular example, the microcontroller 620 may be, for example, the LM 4F230H5QR available from Texas Instruments. In at least one embodiment, the Texas Instruments LM4F230H5QR is an ARM Cortex-M4F processor core that includes, among other features readily available in the product datasheet, 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance beyond 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and one or more 12-bit ADCs with 12 analog input channels. Other microcontrollers may be readily substituted for use with module 4410. Therefore, this disclosure should not be limited to this context.

[0178] Memory 624 may include program instructions for controlling each of the motors of the surgical instrument 600, which can be connected to a common control module 610. For example, memory 624 may include program instructions for controlling the firing motor 602, the closing motor 603, and the joint movement motors 606a, 606b. Such program instructions can cause the processor 622 to control the firing function, closing function, and joint movement function according to input from an algorithm or control program of the surgical instrument or tool.

[0179] For example, one or more mechanisms and / or sensors, such as sensor 630, can be used to alert the processor 622 to program instructions that should be used in a particular setting. For example, sensor 630 can alert the processor 622 to use program instructions related to the firing, closing, and joint movement of the end effector. In a particular example, sensor 630 may include a position sensor that can be used to sense the position of switch 614, for example. Thus, if the processor 622 detects, for example, via sensor 630 that switch 614 is in a first position 616, it can use a program instruction associated with the firing of the end effector's I-beam; if the processor 622 detects, for example, via sensor 630 that switch 614 is in a second position 617, it can use a program instruction associated with the closing of the anvil; and if the processor 622 detects, for example, via sensor 630 that switch 614 is in a third position 618a or a fourth position 618b, it can use a program instruction associated with the joint movement of the end effector.

[0180] Figure 9 shows a diagram of a context-aware surgical system 5100 according to at least one aspect of the present disclosure. In some examples, the data source 5126 may include, for example, a modular device 5102 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 5122 (e.g., an EMR database containing patient records), and a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor). The surgical hub 5104 may be configured to derive contextual information about a surgical procedure from the data, for example, based on a particular combination of received data or a particular order in which data is received from the data source 5126. Contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, a particular step in the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function relating to some aspects of the surgical hub 5104 for deriving or inferring information about a surgical procedure from received data may also be referred to as “context-aware.” In one example, the surgical hub 5104 may incorporate a context-aware system, which is hardware and / or programming associated with the surgical hub 5104, that derives contextual information related to the surgical procedure from the received data.

[0181] The situation-aware system of the surgical hub 5104 can be configured to derive contextual information from data received from the data source 5126 in various different ways. For example, the situation-aware system may include a pattern recognition system or machine learning system (e.g., an artificial neural network) trained on training data to correlate various inputs (e.g., data from the database 5122, patient monitoring device 5124, and / or modular device 5102) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. For example, the situation-aware system may include a lookup table that stores pre-characterized contextual information about the surgical procedure, associated with one or more inputs (or ranges of inputs) corresponding to that contextual information. In response to a query with one or more inputs, the lookup table can return the corresponding contextual information of the situation-aware system to control the modular device 5102. In the example, contextual information received by the situation awareness system of the surgical hub 5104 may be associated with a specific control adjustment of one or more modular devices 5102, or a set of control adjustments. In the example, the situation awareness system may include a further machine learning system, a lookup table, or other such system that generates or retrieves one or more control adjustments of one or more modular devices 5102 when contextual information is provided as input.

[0182] The surgical hub 5104 incorporating a situational awareness system can bring several advantages to the surgical system 5100. One advantage may include improved interpretation of the sensed and collected data, which improves processing accuracy during the course of the surgical procedure and / or the use of the data. Returning to the previous example, the situational awareness surgical hub 5104 can determine what type of tissue is being operated on, so if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situational awareness surgical hub 5104 can correctly accelerate or decelerate the motor of the surgical instrument according to the type of tissue.

[0183] The type of tissue being operated on can affect the adjustments made to the compression rate and load threshold of surgical stapling and cutting instruments for measuring specific interstitial gaps. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic or abdominal surgery, thereby allowing the surgical hub 5104 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is the lung (in the case of thoracic surgery) or the stomach (in the case of abdominal surgery). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument according to the type of tissue.

[0184] The type of body cavity being operated on during an aeration procedure can affect the function of the fume extractor. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing aeration) and determine the type of procedure. Generally, since certain procedure types may be performed in specific body cavities, the surgical hub 5104 can appropriately control the motor speed of the fume extractor to match the body cavity being operated on. Thus, the situational awareness surgical hub 5104 can provide a consistent amount of fume extraction for both thoracic and abdominal surgeries.

[0185] The type of procedure being performed can affect the optimal energy level for operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopy, the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid, which may require a higher energy level. The situational awareness surgical hub 5104 can determine whether the surgical procedure is an arthroscopy. The surgical hub 5104 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level for operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situational awareness surgical hub 5104 can determine what type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue shape for the surgical procedure. Furthermore, the situation-aware surgical hub 5104 can be configured to adjust the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument not simply for each procedure, but throughout the course of the surgical procedure. The situation-aware surgical hub 5104 can determine which steps of the surgical procedure are being performed or are continuing, and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy levels to values ​​appropriate for the expected tissue types according to the surgical procedure.

[0186] In the example, the surgical hub 5104 may also derive data from an additional data source 5126 to improve conclusions drawn from one data source 5126. The contextually aware surgical hub 5104 may enhance data received from the modular device 5102 with contextual information constructed from other data sources 5126 regarding the surgical procedure. For example, the contextually aware surgical hub 5104 may be configured to determine whether hemostasis has been achieved (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, video or image data may not be conclusive. Therefore, in one example, the surgical hub 5104 may be further configured to make a determination regarding the integrity of the staple line or tissue weld by comparing physiological measurements (e.g., blood pressure sensed by a BP monitor communicably connected to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (Figure 2) communicably connected to the surgical hub 5104). In other words, the context-aware system of the surgical hub 5104 can provide additional context when analyzing visualization data by considering physiological measurement data. This additional context can be useful when the visualization data itself may not be conclusive or may be incomplete.

[0187] For example, the situational awareness surgical hub 5104 can proactively activate the generator to which the RF electrosurgical instrument is connected if it determines that the instrument will be needed in a subsequent step of the procedure. By proactively activating the energy source, the instrument can be ready for use as soon as the preceding step of the procedure is completed.

[0188] The situational awareness surgical hub 5104 can determine whether the current or subsequent steps of a surgical procedure require different views or magnifications on the display, according to the characteristics of the surgical site that the surgeon is expected to need to see. The surgical hub 5104 can then appropriately and proactively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108), thereby automatically adjusting the display throughout the surgical procedure.

[0189] The situational awareness surgical hub 5104 can determine which steps of a surgical procedure are being performed or will be performed next, and whether specific data or comparisons of data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the steps of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0190] Errors may be checked during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situation-aware surgical hub 5104 can determine whether the operating room is properly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 may be configured to determine the type of surgical procedure being performed, read the corresponding checklist, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout to a standard layout for the type of surgical procedure that the surgical hub 5104 has determined to be performed. In some examples, the surgical hub 5104 may be configured to compare a list of items for the procedure and / or a list of devices paired with the surgical hub 5104 to a recommended or expected manifest of items and / or devices for a given surgical procedure. If discontinuities exist between the lists, the surgical hub 5104 may be configured to provide an alert indicating that a particular modular device 5102, patient monitoring device 5124, and / or other surgical articles are missing. In some examples, the surgical hub 5104 may be configured to determine the relative distance or relative position of the modular device 5102 and the patient monitoring device 5124, for example, by proximity sensors. The surgical hub 5104 can compare the relative position of the devices to a recommended or predicted layout for a particular surgical procedure. If a discontinuity exists between the layouts, the surgical hub 5104 may be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout.

[0191] The situational awareness surgical hub 5104 can determine whether a surgeon (or other healthcare professional) is making an error or deviating from a set of actions expected during a surgical procedure. For example, the surgical hub 5104 can be configured to determine the type of surgical procedure being performed, read a correspondence list of instrument usage steps or sequences (e.g., from memory), and then compare the steps or instruments being performed or used during the surgical procedure with the steps or instruments expected for the type of surgical procedure that the surgical hub 5104 has determined is being performed. In some examples, the surgical hub 5104 can be configured to provide alerts indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0192] Surgical instruments (and other modular devices 5102) may be adjusted to suit the specific circumstances of each surgical procedure (such as adjustment for different tissue types) and may verify actions during the surgical procedure. Subsequent steps, data, and display adjustments may be provided to the surgical instruments (and other modular devices 5102) in the operating room, according to the specific circumstances of the procedure.

[0193] Figure 10 shows an exemplary surgical procedure timeline 5200 and contextual information that the surgical hub 5104 may derive from data received from data source 5126 at each stage of the surgical procedure. Refer to Figure 9 for the following description of the timeline 5200 shown in Figure 9. The timeline 5200 may illustrate the typical steps that nurses, surgeons, and other healthcare professionals might take during a lung segmentectomy, beginning with setting up the operating room and ending with transferring the patient to the postoperative recovery room. The contextually aware surgical hub 5104 may receive data from data source 5126 throughout the surgical procedure, including data generated each time healthcare professionals use the modular device 5102 paired with the surgical hub 5104. The surgical hub 5104 receives this data from the paired modular device 5102 and other data sources 5126, and can continuously derive estimations (i.e., contextual information) about the procedure in progress as new data is received, such as which step of the procedure is being performed at any given time. The contextual awareness system of the surgical hub 5104 may, for example, record data about the procedure to generate a report, verify the steps being taken by the medical personnel, provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen), adjust the modular device 5102 based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and perform any other such actions described herein.

[0194] As the first step 5202 in this exemplary procedure, hospital staff may retrieve the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 5104 determines that the procedure to be performed is a thoracic surgery. In the second step 5204, staff may scan incoming medical supplies for the procedure. The surgical hub 5104 cross-references the scanned supplies with a list of supplies that may be used in various types of procedures to confirm that the combination of supplies matches a thoracic procedure. Furthermore, the surgical hub 5104 may also determine that the procedure is not a wedge resection (because the incoming supplies either do not include specific supplies required for a thoracic wedge resection or are otherwise not corresponding to a thoracic wedge resection). In the third step 5206, healthcare workers may scan the patient band via a scanner 5128 that is communicably connected to the surgical hub 5104. The surgical hub 5104 can then verify the patient's identity based on the scanned data. In the fourth step 5208, a medical professional turns on the assistive device. The assistive devices used may vary depending on the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, they may include a fume extractor, an inhaler, and a medical imaging device. Once activated, the assistive device, which is a modular device 5102, can automatically pair with a surgical hub 5104, which may be located within a specific vicinity of the modular device 5102, as part of its initialization process. The surgical hub 5104 can then derive contextual information about the surgical procedure by detecting the type of modular device 5102 paired with it during this pre-operative or initialization phase. In this particular example, the surgical hub 5104 can determine that the surgical procedure is a VATS procedure based on this specific combination of paired modular devices 5102. Based on a combination of data from the patient's EMR, a list of medical supplies used in the procedure, and the type of modular device 5102 connected to the hub, the surgical hub 5104 can roughly estimate the specific procedure performed by the surgical team.Once the surgical hub 5104 knows what particular procedure is being performed, it can then read the steps of that procedure from memory or the cloud, and then cross-reference the data subsequently received from connected data sources 5126 (e.g., modular device 5102 and patient monitoring device 5124) to estimate which steps of the surgical procedure the surgical team is performing. In the fifth step 5210, personnel attach the EKG electrode and other patient monitoring devices 5124 to the patient. The EKG electrode and other patient monitoring devices 5124 can be paired with the surgical hub 5104. Once the surgical hub 5104 begins receiving data from the patient monitoring devices 5124, it can confirm that the patient is in the operating room, for example, as described in process 5207. In the sixth step 5212, medical personnel can administer anesthesia to the patient. The surgical hub 5104 can infer that the patient is under anesthesia based on data from the modular device 5102 and / or the patient monitoring device 5124, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 5212 is completed, the preoperative part of the lung segmentectomy is complete and the surgical part begins.

[0195] In the seventh step 5214, the lung of the patient being operated on may collapse (while ventilation is switched to the contralateral lung). The surgical hub 5104 can infer, for example, that the patient's lung has collapsed from the ventilator data. The surgical hub 5104 can compare the detection of the patient's lung collapse with the expected steps of the procedure (which can be accessed or read in advance), so it can infer that the surgical portion of the procedure has started and determine that collapsing the lung may be the first surgical step in this particular procedure. In the eighth step 5216, a medical imaging device 5108 (e.g., a scope) may be inserted and video from the medical imaging device may be started. The surgical hub 5104 can receive medical imaging device data (i.e., video or image data) through the connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 5104 can determine that the laparoscopic portion of the surgical procedure has started. Furthermore, the surgical hub 5104 can determine that a particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that wedge resections have not been taken into consideration by the surgical hub 5104 based on the data received in the second step 5204 of the procedure). Using data from the medical imaging device 124 (Figure 2), contextual information regarding the type of procedure being performed can be determined in various ways, for example, by determining the angle of the medical imaging device directed towards the visualization of the patient's anatomical structure, by monitoring the number or type of medical imaging device being used (i.e., activated and paired with the surgical hub 5104), and by monitoring the type of visualization device being used. For example, one technique for performing a VATS lobectomy positions the camera above the diaphragm in the anteroinferior corner of the patient's thoracic cavity, while another technique for performing a VATS segmentectomy may position the camera in an anterior intercostal position relative to the segmental fissure. The situational awareness system can be trained, for example, using pattern recognition or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. An exemplary technique for performing VATS lobectomy can utilize a single medical imaging device.An exemplary technique for performing VATS partial resection utilizes multiple cameras. One technique for performing VATS segmental resection utilizes an infrared light source (which can be communicably connected to the surgical hub as part of the visualization system) to visualize the segmental fissure, but this is not used in VATS lobectomy. By tracking any or all of this data from the medical imaging device 5108, the surgical hub 5104 can determine the specific type of surgical procedure being performed and / or the technique being used for that specific type of surgical procedure.

[0196] In the ninth step 5218, the surgical team may begin the incision step of the procedure. The surgical hub 5104 receives data from an RF or ultrasound generator indicating that an energy instrument is being emitted, and can therefore infer that the surgeon is in the process of incising and separating the patient's lung. The surgical hub 5104 can cross-reference the received data with the read-out steps of the surgical procedure to determine that the energy instrument being emitted at this point in the process (i.e., after the completion of the steps of the procedure described above) corresponds to the incision step. In the tenth step 5220, the surgical team may proceed to the ligation step of the procedure. The surgical hub 5104 may receive data from surgical stapling and cutting instruments indicating that an instrument is being emitted, and can therefore infer that the surgeon is ligating arteries and veins. As in the previous step, the surgical hub 5104 may derive this inference by cross-referencing the received data from surgical stapling and cutting instruments with the steps in the read-out process. In the eleventh step 5222, the segmentectomy portion of the procedure may be performed. The surgical hub 5104 can infer that a surgeon is transversely incising parenchymal tissue based on data from surgical stapling and cutting instruments (including data from their cartridges). The cartridge data may correspond, for example, to the size or type of staples being fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data may indicate the type of tissue being stapled and / or transversely incised. In this case, the type of staples being fired is used for parenchymal tissue (or other similar tissue types), thereby allowing the surgical hub 5104 to infer that the segmental resection portion of the procedure is being performed. Subsequently, in the twelfth step 5224, the nodule incision step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 5104 can infer that the surgical team is incising the nodule and performing a leak test. In this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue has been transversely incised corresponds to the nodule incision step, thereby allowing the surgical hub 5104 to make this inference.It should be noted that different instruments are better suited to specific tasks, so surgeons should periodically alternate between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the specific stage of the procedure. Therefore, the specific sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which stage of the procedure the surgeon is performing. Once the 12th step 5224 is completed, the incision may be closed and the postoperative portion of the procedure may begin.

[0197] In the 13th step 5226, the patient can be de-anesthetized. The surgical hub 5104 may estimate that the patient is waking up from anesthesia, for example, based on ventilator data (i.e., the patient's respiratory rate begins to increase). Finally, the 14th step 5228 may be the step in which medical personnel remove various patient monitoring devices 5124 from the patient. Thus, the surgical hub 5104 may estimate that the patient is being transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices 5124. As can be seen from this exemplary procedure description, the surgical hub 5104 can determine or estimate when each step of a given surgical procedure is occurring, according to the data received from various data sources 5126 that are communicably connected to the surgical hub 5104.

[0198] As shown in the first step 5202 of the timeline 5200 in Figure 10, in addition to estimating the type of surgical procedure to be performed using patient data from the EMR database, the patient data can also be utilized by the situation-aware surgical hub 5104 to generate control adjustments for the paired modular device 5102.

[0199] Figure 11 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system may be configured to monitor and analyze data relating to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analytics system. The cloud-based analytics system may be described as a surgical system, but is not necessarily limited to that, and may be a cloud-based medical system. As shown in Figure 11, the cloud-based analytics system may include a plurality of surgical instruments 7012 (which may be the same as or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same as or similar to hub 106), and a surgical data network 7001 (which may be the same as or similar to network 201) for connecting the surgical hubs 7006 to a cloud 7004 (which may be the same as or similar to cloud 204). Each of the plurality of surgical hubs 7006 may be communicably connected to one or more surgical instruments 7012. Hub 7006 can also be communicably connected to a cloud 7004 of a computer-implemented interactive surgical system via network 7001. Cloud 7004 may be a remote centralized source of hardware and software for storing, manipulating, and communicating data generated based on the operation of various surgical systems. As shown in Figure 11, access to Cloud 7004 may be achieved via network 7001, which may be the Internet or another suitable computer network. The surgical hub 7006, which can be connected to Cloud 7004, can be considered the client side of a cloud computing system (i.e., a cloud-based analytics system). Surgical instruments 7012 may be paired with the surgical hub 7006 for the control and implementation of various surgical procedures or actions described herein.

[0200] In addition, the surgical instrument 7012 may be equipped with transceivers for transmitting data to and from a corresponding surgical hub 7006 (which may also be equipped with transceivers). The combination of the surgical instrument 7012 and the corresponding hub 7006 can indicate a specific location, such as an operating room within a medical facility (e.g., a hospital) for providing medical surgery. For example, the memory of the surgical hub 7006 can store location data. As shown in Figure 11, the cloud 7004 includes a central server 7013 (which may be the same as or similar to the remote server 7013), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7006. The central server 7013 of the cloud 7004 collectively manages the cloud computing system, which includes monitoring requests from client modules 7006 and managing the processing power of the cloud 7004 to perform those requests. Each central server 7013 may include one or more processors 7008 connected to a suitable memory device 7010, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 7010 may include machine-executable instructions that, when executed, cause the processor 7008 to run a data analysis module 7034 for cloud-based data analysis, operation, recommendations, and other operations described below. Furthermore, the processor 7008 can run the data analysis module 7034 independently or in conjunction with a hub application running independently by the hub 7006. The central server 7013 may also include a database 2212 of aggregated medical data that may reside in memory 2210.

[0201] Based on connections to various surgical hubs 7006 via network 7001, cloud 7004 can aggregate data from various surgical instruments 7012 and specific data generated by their corresponding hubs 7006. Such aggregated data may be stored in the aggregated medical database 7011 of cloud 7004. Specifically, cloud 7004 can advantageously perform data analysis and operations on the aggregated data to derive insights and / or perform functions that individual hubs 7006 cannot achieve on their own. For this purpose, as shown in Figure 11, cloud 7004 and surgical hubs 7006 are communicated together to send and receive information. An I / O interface 7005 is connected to multiple surgical hubs 7006 via network 7001. In this way, the I / O interface 7005 can be configured to transfer information between the surgical hubs 7006 and the aggregated medical data database 7011. Thus, the I / O interface 7005 can facilitate read / write operations of the cloud-based analysis system. Such read / write operations may be performed in response to requests from the hub 7006. These requests may be sent to the hub 7006 via the hub application. The I / O interface 7005 may include one or more high-speed data ports, such as a universal serial bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 7004 to the hub 7006. The hub application server 7002 of the cloud 7004 may be configured to host and supply shared functions to a software application (e.g., a hub application) run by the surgical hub 7006. For example, the hub application server 7002 may manage requests from the hub application through the hub 7006, control access to the aggregated medical data database 7011, and perform load balancing. The data analysis module 7034 will be described in more detail with reference to Figure 12.

[0202] The configurations of the specific cloud computing systems described in this disclosure can be specifically designed to address a variety of problems arising in the context of medical surgeries and procedures performed using medical devices such as surgical instruments 7012, 112. In particular, surgical instrument 7012 may be a digital surgical device configured to interact with the cloud 7004 in order to implement techniques for improving surgical outcomes. Various surgical instruments 7012 and / or surgical hubs 7006 may include a touch-controlled user interface so that a clinician can control the manner of interaction between the surgical instruments 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0203] Figure 12 is a block diagram showing the functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analytics system may include a number of data analytics modules 7034 that can be executed by a processor 7008 of the cloud 7004 to provide data analytics solutions to problems that arise particularly in the medical field. As shown in Figure 12, the functionality of the cloud-based data analytics modules 7034 may be supported via a hub application 7014 hosted by a hub application server 7002 that can be accessed on a surgical hub 7006. The cloud processor 7008 and the hub application 7014 may work together to execute the data analytics modules 7034. An application program interface (API) 7016 may define a set of protocols and routines corresponding to the hub application 7014. In addition, the API 7016 can manage the storage and retrieval of data to and from a medical database 7011 aggregated for the operation of the application 7014. A cache 7018 may also store data (e.g., temporarily) and may be linked to the API 7016 for more efficient retrieval of data used by the application 7014. The data analysis module 7034 in Figure 12 may include modules for resource optimization 7020, data collection and aggregation 7022, authorization and security 7024, control program updates 7026, patient outcome analysis 7028, recommendations 7030, and data classification and prioritization 7032. Other suitable data analysis modules may also be implemented by the cloud 7004 in several embodiments. In one embodiment, the data analysis module may be used for specific recommendations based on the analysis of trends, outcomes, and other data.

[0204] For example, the data collection and aggregation module 7022 may be used to generate self-describing data (e.g., metadata), including the identification of prominent features or configurations (e.g., trends), the management of redundant datasets, and the storage of data into paired datasets that can be grouped by surgery but do not necessarily correspond to actual surgical dates and surgeons. In particular, paired datasets generated from the operation of surgical instruments 7012 may involve applying a binary classification, such as bleeding or non-bleeding events. More generally, the binary classification may be characterized as either a desirable event (e.g., a successful surgical procedure) or an undesirable event (e.g., a surgical instrument 7012 misfired or misused). The aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Thus, the data collection and aggregation module 7022 can generate aggregated metadata or other organized data based on the raw data received from the surgical hub 7006. For this purpose, the processor 7008 can be operationally linked to the hub application 7014 and the aggregated medical data database 7011 in order to execute the data analysis module 7034. The data acquisition and aggregation module 7022 may store the aggregated and organized data in the aggregated medical data database 2212.

[0205] The resource optimization module 7020 can be configured to analyze this aggregated data to determine the optimal use of resources for a particular healthcare facility or group of healthcare facilities. For example, the resource optimization module 7020 can determine the optimal reorder point for a group of healthcare facilities for a surgical instrument 7012 based on the corresponding predicted demand for such instrument. The resource optimization module 7020 can also evaluate the resource use or other operational configurations of various healthcare facilities to determine whether resource use can be improved. Similarly, the recommendation module 7030 can be configured to analyze the aggregated data from the data collection and aggregation module 7022 to provide recommendations. For example, the recommendation module 7030 can recommend to a healthcare facility (e.g., a healthcare service provider such as a hospital) that a particular surgical instrument 7012 should be upgraded to an improved version based, for example, on the error rate being higher than predicted. In addition, the recommendation module 7030 and / or the resource optimization module 7020 can recommend better supply chain parameters, such as product reorder points, and provide suggestions for different surgical instruments 7012, their use, or procedural steps to improve surgical outcomes. Medical facilities can receive such recommendations via the corresponding surgical hub 7006. More specific recommendations regarding the parameters or configurations of various surgical instruments 7012 can also be provided. The hub 7006 and / or the surgical instrument 7012 may each have a display screen that shows data or recommendations provided by the cloud 7004.

[0206] The patient outcome analysis module 7028 can analyze surgical outcomes associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 may also analyze and evaluate other potential operating parameters. In this regard, the recommendation module 7030 may use these other potential operating parameters to make recommendations based on the resulting better surgical outcomes, such as better sealing or less bleeding. For example, the recommendation module 7030 may send a suggestion to the surgical instrument 7006 regarding the use of a particular cartridge with the corresponding stapled surgical instrument 7012. Thus, the cloud-based analytics system may be configured to analyze large amounts of collected raw data while controlling for common variables and to provide centralized recommendations across multiple healthcare facilities (favorably determined based on aggregated data). For example, the cloud-based analytics system can analyze, evaluate, and / or aggregate things such as the type of medical procedure, the type of patient, the number of patients, and geographical similarities among healthcare providers using similar types of instruments in a way that a single healthcare facility cannot analyze independently. The control program update module 7026 can be configured to implement various recommendations for surgical instruments 7012 when the corresponding control program is updated. For example, the patient outcome analysis module 7028 can identify correlations linking specific control parameters to successful (or unsuccessful) outcomes. Such correlations may be addressed when the updated control program is transmitted to the surgical instruments 7012 via the control program update module 7026. Updates to the surgical instruments 7012, which may be transmitted via the corresponding hub 7006, may incorporate aggregated performance data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. In addition, the patient outcome analysis module 7028 and the recommendation module 7030 can identify improved ways of using the surgical instruments 7012 based on the aggregated performance data.

[0207] The cloud-based analytics system may include security features implemented by Cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other preferred security credentials. These credentials may be stored in memory 7010 and associated with permitted cloud access levels. For example, based on providing accurate credentials, a surgical hub 7006 may be granted access to communicate with the cloud to a predetermined extent (e.g., to send or receive certain defined types of information). For this purpose, the aggregated medical data database 7011 of Cloud 7004 may include a database of certified credentials to verify the accuracy of the provided credentials. Different credentials may be associated with various levels of authorization for interaction with Cloud 7004, such as a predetermined access level for receiving data analysis generated by Cloud 7004. Furthermore, for security purposes, the cloud may maintain databases of hubs 7006, surgical instruments 7012, and other devices, which may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be permitted to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be prevented from fully functioning when paired with the corresponding hub 7006. Additionally or alternatively, the cloud 7004 may flag surgical instruments 7012 based on non-conformity or other specified criteria. In this way, counterfeit medical devices and the improper reuse of such devices across the entire cloud-based analysis system can be identified and addressed.

[0208] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal that may represent, for example, authorization credentials for access to the corresponding hub 7006 and the cloud 7004. A wired transceiver can also be used to transmit a signal. Such authorization credentials can be stored in the respective memory devices of the surgical instrument 7012. The authorization and security module 7024 can determine whether the authorization credentials are accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credentials may also be encrypted, for example, by using hash-based encryption. Upon transmitting appropriate authorization, the surgical instrument 7012 may signal to the corresponding hub 7006 and ultimately the cloud 7004 to indicate that the surgical instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may transition to a state in which it can receive medical data for storage in the aggregated medical data database 7011. This readiness for data transmission can be indicated, for example, by an optical indicator on the surgical instrument 7012. Cloud 7004 can also send signals to surgical instruments 7012 to update their associated control programs. Cloud 7004 can send signals directed to specific categories of surgical instruments 7012 (e.g., electrosurgical instruments) to ensure that software updates for control programs are sent only to the appropriate surgical instruments 7012. Furthermore, Cloud 7004 may be used to implement system-wide solutions to address local or global issues based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, Cloud 7004 may modify the authorization credentials corresponding to this group to implement operational lockout for that group.

[0209] A cloud-based analytics system can enable monitoring of multiple healthcare facilities (e.g., healthcare facilities such as hospitals) to determine improved practices and recommend changes accordingly (e.g., via the recommendation module 2030). Thus, the processor 7008 of Cloud 7004 can analyze data associated with individual healthcare facilities to identify facilities and aggregate that data with other data associated with other healthcare facilities. Groups may be defined, for example, based on similar operational practices or geographical location. In this way, Cloud 7004 can provide a wide range of analytics and recommendations to healthcare facility groups. The cloud-based analytics system can also be used for enhanced contextual awareness. For example, the processor 7008 may predictively model the effect of cost and effectiveness recommendations for a particular facility (compared to overall operations and / or various medical procedures). The cost and effectiveness associated with that particular facility can also be compared to the corresponding local areas of other facilities or any other equivalent facilities.

[0210] The data classification and prioritization module 7032 may prioritize and classify data based on severity (e.g., the severity, unexpectedness, or suspiciousness of the medical event associated with the data). This classification and prioritization may be used in conjunction with the functions of other data analysis modules 7034 described herein to improve the cloud-based analysis and operations described herein. For example, the data classification and prioritization module 7032 may assign priorities to data analyses performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels may result in specific responses from the cloud 7004 (corresponding to the level of urgency), such as increased priority for rapid response, special processing, exclusion from the aggregated medical data database 7011, or other preferred responses. Furthermore, if necessary, the cloud 7004 may send requests (e.g., push messages) via the hub application server for additional data from the corresponding surgical instrument 7012. Push messages may result in notifications displayed on the corresponding hub 7006 to request support or additional data. This push message may be needed when the cloud detects a significant anomaly or outlier and the cloud is unable to determine the cause of that anomaly. The central server 7013 can be programmed to trigger this push message in certain critical situations, such as when data is determined to differ from a predicted value by a predetermined threshold, or when security is suspected to be involved.

[0211] Further illustrative details regarding the various functions described are provided in the following description. Each of the various descriptions may utilize a cloud architecture, as shown in Figures 11 and 12 as examples of hardware and software embodiments.

[0212] Figure 13 shows a block diagram of a computer-implemented adaptive surgical system 9060 configured to adaptively generate control program updates for modular devices 9050, according to at least one aspect of the present disclosure. In some examples, the surgical system may include a surgical hub 9000, a plurality of modular devices 9050 communicably connected to the surgical hub 9000, and an analysis system 9100 communicably connected to the surgical hub 9000. While a single surgical hub 9000 may be shown, it should be noted that the surgical system 9060 may include any number of surgical hubs 9000 that can be connected to form a network of surgical hubs 9000 communicably connected to the analysis system 9100. In some examples, the surgical hub 9000 may include a processor 9010 coupled to memory 9020 for executing instructions stored thereon, and a data relay interface 9030 through which data is transmitted to the analysis system 9100. In some examples, the surgical hub 9000 may further include a user interface 9090 having an input device 9092 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 9094 (e.g., a display screen) for providing output to the user. The output may include data from inquiries entered by the user, suggestions of products or mixtures of products for use in a given procedure, and / or instructions for actions to be performed before, during, or after a surgical procedure. The surgical hub 9000 may further include an interface 9040 for communicatively connecting modular devices 9050 to the surgical hub 9000. In one embodiment, the interface 9040 may include transceivers that can be communicatively connected to the modular devices 9050 via a wireless communication protocol. The modular devices 9050 may include, for example, surgical staple fasteners and cutters, electrosurgical instruments, ultrasonic instruments, injectors, respirators, and display screens. In some examples, the surgical hub 9000 may be further connected to one or more patient monitoring devices 9052, such as an EKG monitor or a BP monitor, in a communicative manner.In some examples, the surgical hub 9000 may further be communicatively coupled to one or more databases 9054, such as an EMR database of the medical facility where the surgical hub 9000 is located, or an external computer system.

[0213] When the modular device 9050 is connected to the surgical hub 9000, the surgical hub 9000 can sense or receive pre- and post-operative data from the modular device 9050 and then associate the received pre- and post-operative data with surgical procedure outcome data. The pre- and post-operative data can indicate how the modular device 9050 was controlled during the surgical procedure. The procedure outcome data includes data associated with the outcome of the surgical procedure (or a step thereof), which can include whether the surgical procedure (or a step thereof) had a positive or negative outcome. For example, the outcome data can include whether the patient suffered a post-operative complication from a particular procedure or whether there was a leak (e.g., bleeding or air leak) at a particular staple or incision line. The surgical hub 9000 can obtain surgical procedure outcome data by receiving data from an external source (e.g., from the EMR database 9054), by directly detecting the outcome (e.g., via one of the connected modular devices 9050), or by inferring the occurrence of the outcome through a situation recognition system. For example, data regarding post-operative complications can be read from the EMR database 9054, and data regarding leaks at staple or incision lines can be directly detected or inferred by the situation recognition system. The surgical procedure outcome data can be inferred by a situation recognition system from data received from various data sources, including the modular device 9050 itself, the patient monitoring device 9052, and the database 9054 to which the surgical hub 9000 is connected.

[0214] The surgical hub 9000 can transmit data and result data from associated modular devices 9050 to an analysis system 9100 for processing. By transmitting both pre- and post-operative data and procedure result data that show how the modular devices 9050 are controlled, the analysis system 9100 can correlate different ways of controlling the modular devices 9050 with surgical outcomes of a particular procedure type. In some examples, the analysis system 9100 may include a network of analysis servers 9070 configured to receive data from the surgical hub 9000. Each analysis server 9070 may include memory and a processor coupled to memory that executes instructions stored thereon to analyze the received data. In some examples, the analysis servers 9070 may be connected in a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 9100 can then learn the optimal or preferred operating parameters for various types of modular devices 9050, generate adjustments to the control program of the modular device 9050 in the field, and then transmit (or "push") the updates to the control program of the modular device 9050.

[0215] Further details regarding the computer-implemented interactive surgical system 9060, including the surgical hub 9000 and various modular devices 9050 that can be connected thereto, are described in relation to Figures 5-6.

[0216] FIG. 14 may include a surgical instrument 6502 that provides a surgical system 6500 according to the present disclosure and that can communicate with a console 6522 or a portable device 6526 through a local area network 6518 or a cloud network 6520 via a wired or wireless connection. In various aspects, the console 6522 and the portable device 6526 may be any suitable computing device. The surgical instrument 6502 may include a handle 6504, an adapter 6508, and a loading unit 6514. The adapter 6508 is releasably coupled to the handle 6504, and the loading unit 6514 is releasably coupled to the adapter 6508 such that the adapter 6508 transmits force from a drive shaft to the loading unit 6514. The adapter 6508 or the loading unit 6514 may include a force gauge (not explicitly shown) disposed therein for measuring the force exerted on the loading unit 6514. The loading unit 6514 may include an end effector 6530 having a first jaw 6532 and a second jaw 6534. The loading unit 6514 may be a field loading or multi-firing loading unit (MFLU) that enables a clinician to fire a plurality of fasteners multiple times without the need for the loading unit 6514 to be removed from the surgical site in order to reload the loading unit 6514.

[0217] The first and second jaws 6532, 6534 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 6532 may be configured to fire at least one fastener multiple times or may include a replaceable multi-firing fastener cartridge that includes a plurality of fasteners (e.g., staples, clips, etc.) that may be fired two or more times before being replaced. The second jaw 6534 may include an anvil that deforms or otherwise secures the fastener around the tissue when the fastener is ejected from the multi-firing fastener cartridge.

[0218] The handle 6504 may include a motor connected to the drive shaft so as to affect the rotation of the drive shaft. The handle 6504 may include a control interface for selectively starting the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces that can be engaged by a clinician to start the motor.

[0219] The control interface of the handle 6504 communicates with the controller 6528 of the handle 6504, which can selectively start the motor and affect the rotation of the drive shaft. The controller 6528 may be located within the handle 6504 and is configured to receive input from the control interface and adapter data from the adapter 6508 or loading unit data from the loading unit 6514. The controller 6528 can analyze the input from the control interface and the data received from the adapter 6508 and / or loading unit 6514 to selectively start the motor. The handle 6504 may also include a display that can be viewed by a clinician while the handle 6504 is in use. The display may be configured to show portions of the adapter or loading unit data before, during, or after firing the instrument 6502.

[0220] The adapter 6508 may include an adapter identification device 6510 disposed therein, and the loading unit 6514 may include a loading unit identification device 6516 disposed therein. The adapter identification device 6510 may communicate with the controller 6528, and the loading unit identification device 6516 may communicate with the controller 6528. The loading unit identification device 6516 may communicate with the adapter identification device 6510, which will be understood to relay or pass communication from the loading unit identification device 6516 to the controller 6528.

[0221] Adapter 6508 may also include several sensors 6512 (one shown) disposed around it to detect various conditions of the adapter 6508 or the environment (e.g., when the adapter 6508 is connected to a loading unit, when the adapter 6508 is connected to a handle, when the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside the adapter 6508, the number of times the adapter 6508 fires, the peak force of the adapter 6508 during firing, the total amount of force applied to the adapter 6508, the peak recoil force of the adapter 6508, the number of pauses of the adapter 6508 during firing, etc.). Multiple sensors 6512 can provide input to the adapter identification device 6510 in the form of data signals. The data signals from the multiple sensors 6512 may be stored in the adapter identification device 6510 or used to update adapter data stored in the adapter identification device 6510. The data signals from the multiple sensors 6512 may be analog or digital. Multiple sensors 6512 may include force gauges for measuring the force exerted on the loading unit 6514 during firing.

[0222] The handle 6504 and adapter 6508 may be configured to interconnect the adapter identification device 6510 and the loading unit identification device 6516 with the controller 6528 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). In addition, or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductive transmission). It is also intended that the adapter identification device 6510 and the controller 6528 may be able to wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0223] The handle 6504 may include a transmitter 6506 configured to transmit instrument data from the controller 6528 to other components of the system 6500 (e.g., LAN 6518, cloud 6520, console 6522, or portable device 6526). The transmitter 6506 may also receive data (e.g., cartridge data, loading unit data, or adapter data) from other components of the system 6500. For example, the controller 6528 may transmit instrument data to the console 6522, including the serial number of an attached adapter (e.g., adapter 6508) attached to the handle 6504, the serial number of a loading unit attached to the adapter (e.g., loading unit 6514), and the serial number of a multi-fire fastener cartridge (e.g., multi-fire fastener cartridge) loaded into the loading unit. The console 6522 may then send data (e.g., cartridge data, loading unit data, or adapter data) associated with the attached cartridge, loading unit, and adapter, respectively, back to the controller 6528. The controller 6528 can display a message on the local device display, or transmit a message via the transmitter 6506 to the console 6522 or portable device 6526, which can then display the message on the display 6524 or portable device screen, respectively.

[0224] Figure 15A shows an exemplary flow for determining the operating mode and operating in the determined mode. Computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems may be configured to be updatable. Such updates may include the inclusion of features and benefits that were not available to the user before the update. These updates can be established by any method of hardware, firmware, and software updates suitable for introducing functionality to the user. For example, computer-implemented interactive surgical systems and / or components and / or subsystems of computer-implemented interactive surgical systems can be updated using interchangeable / swappable (e.g., hot-swappable) hardware components, flashable firmware devices, and updatable software systems.

[0225] Renewals may be conditional on any preferred criteria or set of criteria. For example, a renewal may be conditional on one or more hardware capabilities of the system, such as processing power, bandwidth, or resolution. For example, a renewal may be conditional on one or more software aspects, such as the purchase of specific software code. For example, a renewal may be conditional on a purchased service tier. A service tier may represent features and / or sets of features that a user is entitled to use in conjunction with a computer-implemented interactive surgical system. A service tier may be determined by a license code, an e-commerce server authentication interaction, a hardware key, a username / password combination, a biometric authentication interaction, a public / private key exchange interaction, etc.

[0226] In 10704, system / device parameters may be identified. System / device parameters may be any element or set of elements on which updates are conditional. For example, a computer-implemented interactive surgical system may detect a certain bandwidth of communication between a modular device and a surgical hub. For example, a computer-implemented interactive surgical system may detect an instruction to purchase a certain service layer.

[0227] In 10708, the operating mode may be determined based on identified system / device parameters. This determination may be made by a process that maps system / device parameters to operating modes. The process may be manual and / or automated. The process may be the result of local and / or remote computing. For example, a client / server interaction may be used to determine the operating mode based on identified system / device parameters. For example, local software and / or locally embedded firmware may be used to determine the operating mode based on identified system / device parameters. For example, a hardware key, such as a secure microprocessor, may be used to determine the operating mode based on identified system / device parameters.

[0228] In 10710, operation may proceed according to a determined operating mode. For example, the system or device may proceed to operate in the default operating mode. For example, the system or device may proceed to operate in an alternative operating mode. The operating mode may be indicated by control hardware, firmware, and / or software already residing in the system or device. The operating mode may be indicated by newly installed / updated control hardware, firmware, and / or software.

[0229] Figure 15B shows an exemplary functional block diagram for changing the operating mode. The upgradeable element 10714 may include an initialization component 10716. The initialization component 10716 may include any hardware, firmware, and / or software suitable for determining the operating mode. For example, the initialization component 10716 may be part of a system or device startup procedure. The initialization component 10716 may participate in interactions to determine the operating mode of the upgradeable element 10714. For example, the initialization component 10716 may interact with, for example, a user 10730, an external resource 10732, and / or a local resource 10718. For example, the initialization component 10716 may receive a license key from user 10730 to determine the operating mode. The initialization component 10716 may query an external resource 10732, such as a server, using the serial number of the upgradeable device 10714 to determine the operating mode. For example, the initialization component 10716 may query the local resource 10718, such as a local query to determine the amount of available bandwidth and / or a local query for a hardware key to determine, for example, the operating mode.

[0230] An upgradeable element 10714 may include one or more operational components 10720, 10722, 10726, 10728 and an operational pointer 10724. The initialization component 10716 can instruct the operational pointer 10724 to direct the operation of the upgradeable element 10741 to the operational components 10720, 10722, 10726, 10728 corresponding to the determined operational mode. The initialization component 10716 can instruct the operational pointer 10724 to direct the operation of the upgradeable element to the default operational component 10720. For example, the default operational component 10720 may be selected on the condition that no other alternative operational modes have been determined. For example, the default operational component 10720 may be selected on the condition of initialization component failure and / or interaction failure. The initialization component 10716 can instruct the operation pointer 10724 to instruct the resident operation component 10722 to perform the operation of the upgradeable component 10714. For example, certain functions may reside within the upgradeable component 10714 but require activation to function. The initialization component 10716 can instruct the operation pointer 10724 to instruct the operation of the upgradeable component 10714 to install a new operation component 10728 and / or a newly installed operation component 10726. For example, new software and / or firmware may be downloaded. The new software and / or firmware may contain code to enable the features represented by the selected operating mode. For example, a new hardware component may be installed to enable the selected operating mode.

[0231] Coordination between a primary display and / or a secondary display may be provided. For example, coordination between a local instrument display and a paired imaging device display may be provided.

[0232] An instrument may be provided that includes a local display and a hub having a separate operating room (OR) or operating room display from the instrument display. When the instrument is connected to the surgical hub, the secondary display on the instrument is reconfigured to display different information than when it is independent of the surgical hub connection. Some of the information on the instrument's secondary display may be displayed on the primary display of the surgical hub. Image fusion may be performed which may allow overlaying one or more of the following: instrument status, integrated landmarks used to interlock several images, and guidance features. Image fusion may be provided on the surgical hub and / or instrument display. As disclosed herein, several techniques may be used to overlay or extend images and / or text from multiple image / text sources in order to present a composite image on one or more displays.

[0233] Collaboration between one or more local instrument displays and a paired laparoscopic display may be provided. The behavior of the instrument's local display may change when sensing the presence of a connectable display (e.g., a global display) that can be linked to a surgical hub. This disclosure may provide a 360° composite upper view of the surgical site that can help avoid collateral structures.

[0234] During a surgical procedure, the surgical site can be displayed on a remote surgical hub display. The remote surgical hub display may be referred to as a primary display. During a surgical procedure, a surgical device can track and record surgical data and variables (e.g., surgical parameters) stored within the instrument (see FIGS. 1-13 for instrument architectures including a processor, memory, control circuitry, storage devices, etc.). Surgical procedure parameters can include, for example, firing force (FTF), force-to-close (FTC), firing progression, tissue gap, power level, impedance, tissue compression stability (creep), etc. Providing an image / text overlay can be provided, for example, to enable a surgeon to view a display presenting the overlaid image / text information.

[0235] When a surgical device (e.g., an instrument) is connected to a surgical hub, a composite image that can include a view of the surgical site received from a first instrument (e.g., a medical imaging device such as a laparoscope, endoscope, thoracoscope, etc.) can be displayed on the primary display, which can be augmented by surgical data and variables received from a second instrument (e.g., a surgical stapler) to provide related images and data on the primary display.

[0236] During a surgical procedure, the surgical site can be displayed as a narrow view of a medical imaging device on a primary surgical hub display. Articles outside the current view, i.e., collateral structures, cannot be seen without moving the medical imaging device.

[0237] One embodiment can provide a narrow view of the surgical site within a first window of a display, augmented by a wide view of the surgical site within a separate window of the display. This provides a composite overhead view mapped using two or more imaging arrays and provides an expanded image of multiple perspective views of the surgical site.

[0238] In one embodiment, a wide field of view of the surgical site can be provided on a first display, which may be a primary display. A narrower field of view of the surgical procedure side can be provided on a second display, which may be a secondary display.

[0239] A surgical hub may be provided, which may include a processor and memory coupled to the processor. The memory may store instructions executable by the processor to detect surgical device connections to the surgical hub, send control signals to detected surgical devices to transmit surgical parameter data related to the detected devices to the surgical hub, receive surgical parameter data, receive image data from an image sensor, and display the images received from the image sensor, along with the surgical parameter data received from the surgical device, on a display coupled to the surgical hub.

[0240] In another embodiment, the Disclosure provides a surgical hub comprising a processor and memory coupled to the processor. The memory may store instructions executable by the processor to receive first image data from a first image sensor, receive second image data from a second image sensor, and display a first image corresponding to a first field of view and a second image corresponding to a second field of view on a display coupled to the surgical hub. The first image data represents a first field of view, and the second image data represents a second field of view. The display may be a primary display and / or a secondary display. The display may be inside or outside a sterile field.

[0241] The first field of view may be a narrow-angle field of view, and the second field of view may be a wide-angle field of view. The first image may be augmented on a display using the second image. The first image may be fused with the second image to form a third image, which can then be displayed on the display. The fused image data may include instrument data, such as status information associated with the surgical device, image data integration landmarks for interlocking multiple images, and guidance parameters. The first image sensor may capture first image data at a first time and second image data at a second time.

[0242] A third image sensor can receive a third image data, which can represent a third field of view. A composite image data including the second and third image data can be generated. The first image may be displayed on a first display and / or within a first window of the display. The first image may correspond to the first image data. The third image may be displayed on a second display and / or within a second window of the first display. The third image may correspond to the composite image data. The displays may be a primary display and / or a secondary display. This display may be inside or outside the sterile field.

[0243] The third image data may represent a third field of view. The second image data may be fused with the third image data to generate fused image data. The first image may be displayed on a first display and / or within a first window of the display. The first image may correspond to the first image data. The third image may be displayed on a second display and / or within a second window of the first display. The third image may correspond to the fused image data.

[0244] Displaying endoscopic images augmented with surgical instrument images on a primary surgical hub display may allow surgeons to focus on the display and obtain a field of view of the surgical site augmented with surgical instrument data associated with the surgical procedure, such as firing force, closing force, firing progression, interstitial space, power level, impedance, and tissue compressive stability (creep). Endoscopic images may be augmented with surgical instrument images and displayed on the primary and / or secondary displays. For example, the primary display may display an endoscopic image augmented with surgical instrument images, and the secondary display may display surgical instrument images. As described herein, users may use gestures and / or issue commands to change the primary and / or secondary displays. For example, a user may move an image displayed on the secondary display to the primary display, and vice versa. By displaying a narrow-field image in the first window of the display and a composite image of several other perspectives, such as a wider field of view, surgeons can view a magnified image of the surgical site simultaneously with a wider field of view of the surgical site without moving the scope.

[0245] Both a global and a local display may be provided for the device (e.g., a surgical instrument). The local display may be linked to a surgical hub. The global display may be associated with a primary display. The local display may be associated with a secondary display. The device may display one or more (e.g., all) of its associated menus on the local display until it senses a connection to the surgical hub, at which point a subset of the information may be displayed on the primary display, e.g., a monitor, through the surgical hub. The information may or may not be mirrored on the device display. The information may be removed from the device screen. This technique frees up the device display to show different information on the surgical hub display or to display information in a larger font.

[0246] The instrument may have a local display, which may be a secondary display. The surgical hub may be separate from the instrument display and may be associated with a room (e.g., room or OR) display, which may be a primary display. When the instrument is connected to the surgical hub, the instrument local display may become a secondary display, and the instrument may be reconfigured to display different information than when it is operating independently of the surgical hub connection. In another embodiment, some of the information on the secondary display may be displayed on a room primary display via the surgical hub.

[0247] Figure 16 shows a primary display of a surgical hub. For example, Figure 16 shows an exemplary primary display 6200 associated with a surgical hub 206 having a global display window 6202 and a local instrument display window 6204, according to one aspect of the present disclosure. Continuing to refer to Figures 1 to 11, which show interaction with an interactive surgical system 100 environment including surgical hubs 106, 206, and Figures 12 to 14, which show instruments connected to the surgical hub together, the behavior of the local instrument display 6204 can be displayed when an instrument 235 senses the connectable presence of the global display window 6202 through the surgical hub 206. The global display window 6202 may, for example, show a field of view 6206 of a surgical site 6208 as seen through a medical imaging device, such as a laparoscope / endoscopy 219 coupled to an imaging module 238, at the center of a surgical hub display 215, also referred to herein as a monitor. The end effector portion 6218 of the connected instrument 235 may be shown within the field of view 6206 of the surgical site 6208 in the global display window 6202. The image shown on the display 237 located on the instrument 235 connected to the surgical hub 206 is shown, for example, on the local instrument display window 6204 located in the lower right corner of the monitor 6200, or mirrored, as shown in Figure 16.

[0248] During operation, the relevant instruments, information, and menus may be displayed on the display 237 located on the instrument 235 until the instrument 235 senses its connection to the surgical hub 206, at which point all or some subsets of the information presented on the instrument display 237 may be displayed (for example, only) on the local instrument display window 6204 portion of the surgical hub display 6200 via the surgical hub 206. The information displayed on the local instrument display window 6204 may be mirrored on the display 237 located on the instrument 235, or may no longer be accessible on the startup screen of the instrument display 237. This technique releases the instrument 235 to display different information or larger font information on the surgical hub display 6200.

[0249] The primary display 6200 may provide pre- and post-operative visualization of the surgical site 6208. Advanced imaging may identify and visually highlight (6222) important structures such as the ureter 6220 (or nerves, etc.), which may be tracked by the instrument proximity display 6210, shown to the left of the display 6200. In the illustrated example, the instrument proximity display 6210 may show instrument-specific settings. For example, the upper instrument proximity display 6212 may show settings for a unipolar instrument, the intermediate instrument proximity display 6214 may show settings for a bipolar instrument, and the lower instrument proximity display 6212 may show settings for an ultrasound instrument.

[0250] One or more secondary displays, which may be dedicated local displays, may be linked to the surgical hub 206 to provide both an interactive portal via a touchscreen display and / or secondary screens capable of displaying tracked data feeds of any number of surgical hubs 206 to provide status. The secondary screens may display firing force (FTF), interstitial space, power level, impedance, tissue compressive stability (creep), etc., while the primary screen may display key variables (e.g., key variables only) to keep the feed clutter-free. The interactive displays can be used to move the display of information on the primary display to a desired position, size, color, etc. For example, a user can use the interactive displays to move information to the primary display, where it can be highlighted and / or displayed to stand out from other data.

[0251] As shown in Figure 16, the secondary screen displays the instrument proximity display 6210 to the left of the display 6200 and the local instrument display 6204 to the lower right of the display 6200. The local instrument display 6204 presented on the surgical hub display 6200 displays icons for the end effector 6218, for example, the icon for the staple cartridge 6224 currently in use, the size 6226 (e.g., 60 mm) of the staple cartridge 6224, and the icon for the current position of the knife 6228 of the end effector.

[0252] A display 237 located on the instrument 235 can display the wireless or wired mounting of the instrument 235 to the surgical hub 206, and the instrument's communications / records on the surgical hub 206. Settings may be provided on the instrument 235 to allow the user to choose to mirror or extend the display on both monitoring devices. An instrument control unit may be used to interact with the surgical hub display of the information supplied on the instrument. As disclosed herein, the instrument 235 may include a wireless communication circuit for wirelessly communicating with the surgical hub 206.

[0253] A first instrument connected to the surgical hub 206 may be paired with the screen of a second instrument connected to the surgical hub 206, allowing both instruments to display some hybrid combination of information from both devices, becoming a mirror image of the primary display portion.

[0254] The primary display 6200 of the surgical hub 206 can provide a 360° composite top view of the surgical site 6208 to avoid collateral structures. For example, a secondary display of an end-effector surgical stapler may be provided within the primary display 6200 of the surgical hub 206, or on a separate display, to provide a better view around the area within the current field of view 6206.

[0255] Figure 17 shows an example of a primary display for a surgical hub. For example, Figure 17 may show an exemplary primary display having time to provide a composite overhead view of the end effector 6234 portion of a surgical stapler mapped using two or three or more imaging arrays or one array, and multiple perspective views of the end effector 6234 to enable composite imaging of the overhead view. The techniques described herein may be applied to ultrasound instruments, electrosurgical instruments, combinations of ultrasound / electrosurgical instruments, and / or combinations of surgical staplers / electrosurgical instruments. Several techniques may be performed to overlay or extend images and / or text from multiple image / text sources in order to present a composite image on a display (e.g., a single display).

[0256] As shown in Figure 17, the primary display 6200 of the surgical hub 206 can display a primary window 6230. The primary window 6230 may be located in the center of the screen and shows an enlarged or exploded narrow-angle view of the surgical field 6232. The primary window 6230 located in the center of the screen shows an enlarged or narrow-angle view of the end effector 6234 of a surgical stapler that grasps a blood vessel 6236. The primary window 6230 can display an organized image to generate a composite image that enables visualization of structures adjacent to the surgical field 6232. A second window 6240 may be located in the lower left corner of the primary display 6200. The second window 6240 displays an organized image in a wide-angle view with standard focus on the image shown in the primary window 6230 in an overhead view. The overhead view provided in the second window 6240 may allow the observer to easily view items outside the narrow-field surgical field 6232 without moving the laparoscope or other imaging device coupled to the imaging module 238 of the surgical hub 206. A third window 6242 may be shown in the lower right corner of the primary display 6200 and shows an icon 6244 representing the staple cartridge of the end effector 6234 (for example, a staple cartridge in this example), along with additional information such as "4 rows" indicating the number of staple rows 6246 and "35 mm" indicating the distance 6248 the knife traverses along the length of the staple cartridge. Below the third window 6242 is an icon 6258 of the frame of the current state of the clamp stabilization sequence 6250, which indicates clamp stabilization.

[0257] In an exemplary visualization control mode, the display may be controlled by the user, for example, via motion tracking (e.g., head orientation relative to the monitor), hand gestures, voice activation, and other means within the sterile field. The user may use gestures, motion tracking commands, voice activation, etc., to move data from one display to another. For example, the user may use a gesture to move data from a first display to a second display. The gesture may be detected by a hub, which may command the first display to delete the data or stop displaying the data, or command the second display to display the data.

[0258] Figure 18 shows four wide-angle view images of the surgical site at four distinct time points during the procedure. For example, Figure 18 shows Figure 6270 of four distinct wide-angle view images 6272, 6274, 6276, and 6278 of the surgical site at four distinct time points during the procedure, according to one aspect of the present disclosure.

[0259] The sequence of images demonstrates the creation of overhead composite images at wide and narrow focal points over time. The first image 6272 is a wide-angle view of the end effector 6234 clamping the vessel 6236, taken at a preceding time t0 (e.g., 09:35:09). The second image 6274 is another wide-angle view of the end effector 6234 clamping the vessel 6236, taken at the current time t1 (e.g., 09:35:13). The third image 6276 is a composite image of an overhead view of the end effector 6234 clamping the vessel 6236, taken at the current time t1. The third image 6276 may be displayed in the second window 6240 of the primary display 6200 of the surgical hub 206, as shown in Figure 17. The fourth image 6278 is a narrow-angle view of the end effector 6234 clamping the blood vessel 6236 at the current time t1 (e.g., 09:35:13). The fourth image 6278 is a narrow-angle view of the surgical site shown in the primary window 6230 of the primary display 6200 of the surgical hub 206, as shown in Figure 17.

[0260] In one aspect of this disclosure, the primary display and / or secondary display may display one or more of the first, second, third, and / or fourth images. For example, the primary display may display the third image and the secondary display may display the fourth image. Alternatively, the primary display may display the fourth image and the second display may display the third image.

[0261] Figure 19 shows an example of an augmented video image of a preoperative video image augmented with data that identifies the displayed elements. The preoperative video image that can be augmented with data may be displayed on a primary display and / or a secondary display. For example, the augmented video image may be displayed on the primary display and the video image on the secondary display. Alternatively, the augmented video image may be displayed on the secondary display, while the video image may be displayed on the primary display.

[0262] For example, Figure 19 shows an example of an augmented video image 6350 that includes a preoperative video image 6352 augmented with data (e.g., 6354, 6356, 6358 that identify the displayed elements). An augmented reality visual system may be employed in a surgical procedure to implement a method for augmenting data onto the preoperative image 6352. This method includes generating a preoperative image 6352 of the patient's anatomical divisions and generating an augmented video image of the surgical site within the patient. The augmented video image 6350 may include images of at least a portion of a surgical tool 6354 operated by a user 6456. This method may further include processing the preoperative image 6352 to generate data relating to the patient's anatomical divisions. The data may include labels 6358 of the anatomical divisions and peripheral margins of at least a portion of the anatomical divisions. The peripheral margins can be configured to guide the surgeon to the cutting position relative to the anatomical segment, embed data and user identification information within the preoperative image 6350, and display an augmented video image 6350 of the patient's anatomical segment to the user. The method may further include sensing the load state on the surgical tool 6354, generating a feedback signal based on the sensed load state, and updating the data and location of the user identification information embedded within the augmented video image 6350 in real time in response to changes in the location of the surgical tool 6354 within the augmented video image 6350. Further examples are disclosed in U.S. Patent No. 9,123,155, “APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES,” issued 1 September 2015, which is incorporated herein by reference in its entirety.

[0263] In one embodiment, radiographic integration technology can be used to overlay the preoperative image 6352 with data acquired through live internal sensing or pre-procedure technology. Radiographic integration may include identification of surgical landmarks, markers and landmarks using radiographic markers placed inside or outside the patient, and identification of radiopaque staples, clips, or other tissue fixation items. Digital radiographic technology can be used to generate digital images for overlaying with the preoperative image 6352. Digital radiographic imaging is a form of radiography that uses a digital image capture device having a digital radiographic sensor instead of conventional photographic film. Digital radiographic technology provides immediate image preview and availability for overlaying with the preoperative image 6352. Furthermore, special image processing techniques can be applied to the digital radiographic image to improve the overall display quality of the image.

[0264] Digital X-ray imaging techniques may employ image detectors, including flat panel detectors (FPDs), which may be classified into two categories: indirect FPDs and direct FPDs. Indirect FPDs may include amorphous silicon (a-Si) combined with a scintillator in the outer layer of the detector, fabricated from cesium iodide (CSI) or gadolinium oxysulfide (Gd2O2S), which converts X-rays into light. The light may be guided through an a-Si photodiode layer, where it is converted into a digital output signal. The digital signal may then be read out by a thin-film transistor (TFT) or a fiber-coupled charge-coupled device (COD). Direct FPDs include amorphous selenium (a-Se) FPDs that directly convert X-ray photons into charges. In this design, the outer layer of the flat panel may be a high-voltage bias electrode. X-ray photons may generate electron-hole pairs within the a-Se, and the passage of these electrons and holes may depend on the potential of the bias voltage charge. Since holes can be replaced by electrons, the resulting charge pattern within the selenium layer can be read out by a TFT array, an active matrix array, a voltmeter probe, or microplasma line addressing. Other direct digital detectors may be based on CMOS and CCD technology. Phosphorescent detectors may also be used to record X-ray energy during exposure and may be scanned by laser diodes to excite stored energy which may be emitted and read out by a digital image capture array of a CCD.

[0265] Figure 20 shows an exemplary flowchart of a process for displaying one or more images. For example, Figure 20 shows a logical flowchart 6360 of a process showing a control program or logical configuration for displaying images, according to one aspect of the present disclosure. Referring also to Figures 1 to 11 to illustrate interaction with an interactive surgical system 100 environment including surgical hubs 106, 206, the present disclosure provides, in one aspect, a surgical hub 206 comprising a processor 244 and a memory 249 coupled to the processor 244. The memory 249 stores instructions that can be executed by the processor 244 to receive first image data from a first image sensor (6362), receive second image data from a second image sensor (6364), and display on a display a first image corresponding to a first field of view and a second image corresponding to a second field of view (6366). The first image data may represent a first field of view, and the second image data may represent a second field of view. The display may be a primary display and / or a secondary display. The display may be a display 217 connected to a surgical hub 206.

[0266] In one embodiment, the first field of view may be a narrow-angle field of view, and the second field of view may be a wide-angle field of view. In another embodiment, memory 249 stores instructions that can be executed by processor 244 to extend the first image on the display with the second image. The display may be a primary display and / or a secondary display.

[0267] In another embodiment, memory 249 stores instructions that can be executed by processor 244 to merge a first image and a second image into a third image and to display the fused image on a display. The display may be a primary display and / or a secondary display. The display may be display 217. The first image, the second image, and / or the third image may be displayed on the secondary display, while the fused image may be displayed on the primary display. The first image, the second image, and / or the third image may be displayed on the primary display, while the fused image may be displayed on the secondary display.

[0268] In another embodiment, the fused image data includes status information associated with the surgical device 235, an image data integration landmark for interlocking multiple images, and at least one guidance parameter. In another embodiment, the first image sensor is the same as the same image sensor, the first image data is captured at a first time, and the second image data is captured at a second time. One or more images may be displayed on a primary display and / or a secondary display.

[0269] In another embodiment, memory 249 stores instructions that can be executed by processor 244 to receive third image data from third image sensor (the third image data represents a third field of view), generate composite image data including second and third image data, display first image in first window of display (the first image corresponds to first image data), and display third image in second window of display 215 (the third image corresponds to composite image data). In another embodiment, the first image, second image, and / or third image may be displayed on primary and / or secondary displays. For example, the user may indicate that primary and / or secondary displays may display one of the first, second, and third images.

[0270] In another embodiment, memory 249 stores instructions that can be executed by processor 244 to receive third image data from third image sensor (the third image data representing a third field of view), fuse the second and third image data to generate fused image data, display the first image in the first window of display 217 (the first image corresponds to the first image data), and display the third image in the second window of display 217 (the third image corresponds to the fused image data). In another embodiment, the first image, the second image, and / or the third image may be displayed on primary and / or secondary displays. For example, the user may indicate that primary and / or secondary displays may display one of the first, second, and third images.

[0271] In one embodiment, the present disclosure provides and illustrates a surgical communication and control headset that interfaces with a surgical hub 206, as described in relation to Figures 1 to 11. Further examples are disclosed in U.S. Patent Application Publication No. 2009 / 0046146, “SURGICAL COMMUNICATION AND CONTROL SYSTEM,” published on February 19, 2009, which is incorporated herein by reference in its entirety. Figure 21 shows a schematic diagram of a beam source and combined beam detector system used as an apparatus control mechanism in an operating room, according to at least one embodiment of the present disclosure. For example, Figure 21 shows a schematic diagram of a beam source and combined beam detector system used as an apparatus control mechanism in an operating room. System 6680 may be configured and wired to enable apparatus control using an overlay generated on a primary display (e.g., a primary procedure display) and / or a secondary display. A foot switch shows how a user can click a command icon that appears on a screen while the beam source is being used to aim at a specific desired command icon to be clicked. The beam source can also be used to indicate where the user is looking. The beam source can also be used by the user to indicate where data may be displayed. For example, the user may point the beam source at a primary and / or secondary display to indicate which display should be used to view the data.

[0272] The control system's graphic user interface (GUI) communicates with the device control processor, and parameters are changed using the system. The system may include a display that can be coupled to a beam detection sensor. The display may be a primary display and / or a secondary display. For example, system 6680 includes a display 6684 coupled to a beam detection sensor 6682. The system may include a head-mounted source 6686. The beam detection sensor 6682 can communicate with the control system GUI overlay processor and the beam source processor 6688. The surgeon can operate a foot switch 6692 or other auxiliary switches that provide signals to the device control interface unit 6694.

[0273] System 6680 can provide sterile clinicians with a means to control procedural equipment in an easy, rapid, and hands-free, centralized manner. The ability to maximize surgical efficiency and minimize the time patients are under anesthesia is crucial for the best patient outcomes. It is common for surgeons, cardiologists, or radiologists to verbally request adjustments to certain medical devices and electronic equipment used in procedures outside the sterile field. To give a few examples, it is common to have to rely on other staff members to make the necessary adjustments to the settings of devices such as cameras, bobbies, surgical beds, shavers, inhalers, and injectors. In many situations, having to order staff members to change settings can delay the procedure because non-sterile staff members are busy with other tasks. Because sterile physicians cannot adjust non-sterile equipment without compromising sterility, they often have to wait for non-sterile staff members to make the requested adjustments to certain devices before resuming the procedure.

[0274] System 6680 allows the user to regenerate a pointer overlay coupled with a GUI using a beam source and beam detector, and enables a simultaneous switching method (i.e., a foot switch, etc.) to allow the clinician to click commands on the primary and / or secondary displays. In one embodiment, the GUI can appear on the procedure video display, which may be the primary and / or secondary display, when activated, such as when the user tilts their head twice to wake up, or when they step on a foot switch provided in the system. Alternatively, a gesture such as tilting the head to the right may activate the system, and another gesture such as tilting the head to the left may simply activate the beam source. When the overlay (referred to as the device control GUI overlay) appears on the screen, the overlay may display button icons representing various surgical devices, and the user can aim at the button icons using the beam source, in this case a laser beam. When the laser is over the appropriate button icon, a foot switch or other simultaneous switching method can be activated, which acts effectively like a mouse click on a computer. For example, the user can "wake up" the system to pop up a device control GUI overlay, which lists button icons on the screen, each labeled as a corresponding medical device for treatment. The user may make a selection by pointing the laser at the correct box or device and clicking the foot pedal (or some other parallel control such as voice control or a waistband button), similar to clicking a mouse on a computer. The sterile physician can then select, for example, "Inhaler." The next screen shows arrow icons that can be clicked for various settings of the device that need to be adjusted (pressure, speed, etc.). In one iteration, the user can then point the laser at the upward arrow and repeatedly click the foot pedal until the desired setting is achieved.

[0275] In one embodiment, a user, such as a sterile physician, may use a beam to indicate where data may be displayed. For example, the user may be able to view a primary display and / or a secondary display. The user may want to view contextual data, such as operation-related data, on one or more of the displays. The user may use a beam to indicate that contextual data should appear on the primary display. The user may use a beam to indicate that contextual data should appear on the secondary display. The user may use a beam to indicate that data should be moved from the primary display to the secondary display, or vice versa.

[0276] The surgical hub may provide interface control to one or more primary displays and / or secondary displays, which may be secondary surgical display units. The primary and / or secondary displays may be designed to be located within a sterile field.

[0277] Figures 22A to 22E illustrate various types of sterile field control and data entry consoles according to at least one aspect of the present disclosure. Figure 22A shows a single-zone sterile field control and data entry console. Figure 22B shows a multi-zone sterile field control and data entry console. Figure 22C shows a tethered sterile field control and data entry console. Figure 22D shows a battery-powered sterile field control and data entry console. Figure 22E shows a battery-powered sterile field control and data entry console.

[0278] In one embodiment, the surgical hub 206 may provide a secondary user interface that enables the display and control of the functions of the surgical hub 206 from a sterile field. The secondary display may be used to change the display position, which information is displayed where, and to pass off the control of a particular function or device. For example, the secondary display may be used by the user to move a data display on the secondary display to the primary display. In another example, the secondary display may be used by the user to move data from the primary display to the secondary display. The secondary display may be inside the medical device, outside the medical device, or associated with the medical device.

[0279] A display unit, which may be a primary and / or secondary display, may be designed for use within a sterile field and may be accessible for input and display by the surgeon, allowing the surgeon to have interactive input control from the sterile field to control other surgical instruments that may be connected to a surgical hub. The display unit may be sterile, located within the sterile field, and allow the surgeon to interface with the display unit, and the surgical hub to interface directly with and configure instruments as needed, without leaving the sterile field. The display unit may be used for display, control, and tool control exchange, allowing the surgeon to receive supplies from other surgical hubs without leaving the sterile field. The display unit may allow a user, such as a surgeon, to control a primary and / or secondary display that may be outside the sterile field. The display unit may allow a user to control a primary and / or secondary display that may be within the sterile field.

[0280] In one embodiment, the disclosure provides a control unit comprising an interactive touchscreen display, an interface configured to connect the interactive touchscreen display to a surgical hub, a processor, and memory connected to the processor. The memory stores instructions executable by the processor to receive input commands from the interactive touchscreen display located inside the sterile field, transmit the input commands to the surgical hub, and control a device connected to the surgical hub located outside the sterile field.

[0281] In one embodiment, the disclosure provides a control unit comprising an interactive touchscreen display, an interface configured to connect the interactive touchscreen display to a surgical hub, and a control circuit configured to receive input commands from the interactive touchscreen display located inside the sterile field, transmit the input commands to the surgical hub, and control a device connected to the surgical hub located outside the sterile field.

[0282] A display unit may be provided that can be used within a sterile field and may be accessible to the surgeon for input and display. For example, the display unit may provide the surgeon with interactive input control from the sterile field to control other surgical devices connected to a surgical hub.

[0283] This display unit within the sterile field is sterile and allows the surgeon to interface with it and the surgical hub. This gives the surgeon control of instruments coupled to the surgical hub and allows the surgeon to interface with and configure instruments directly as needed without leaving the sterile field. The display unit may also be used for display, control, and tool control exchange, allowing the surgeon to receive supplies from other surgical hubs without leaving the sterile field. For example, the display unit may be a primary display and / or a secondary display, and the display unit may be used to control the display of data on another primary display and / or secondary display. In another example, the display unit may be used to move data displayed on one display to another display.

[0284] A secondary user interface may be used to enable the display and control of surgical hub functions from within the sterile field. This control unit may be a primary and / or secondary display, and may be a display device such as an iPad, e.g., a portable interactive touchscreen display device configured to be introduced into the operating room in a sterile manner. It may be paired like any other device, or it may be location-sensitive. The display device may be enabled to function in this manner whenever the display device is placed over a location (e.g., a specific location). For example, the display device may be enabled to function in this manner whenever the display device is placed over a draped area of ​​the patient's abdomen during a surgical procedure.

[0285] In one embodiment, the disclosure provides a secondary user interface for enabling the display and control of surgical hub functions from within a sterile field. In one embodiment, the secondary display may be used to change the display position, determine which information is displayed where, and pass off control of a particular function or device. For example, the secondary display may be used to transmit data displayed on the primary display.

[0286] Several different types of secondary surgical displays may exist. For example, one type of secondary display may be designed for use within a sterile field and may be accessible for input and display by the surgeon within a sterile field interactive control display. The sterile field interactive control display may be a shared or common sterile field input control display. The sterile field display may be a primary display and / or a secondary display.

[0287] The sterile field display may be mounted on an operating table, on a stand, or simply placed on the patient's abdomen or chest. The sterile field display is sterile and allows the surgeon to interface with the sterile field display and surgical hub. This may give the surgeon control of the system and allow the surgeon to interface and configure the sterile field display as needed. The sterile field display may be configured as a master device and may be used for display, control, tool control exchange, feeding from other surgical hubs, etc. For example, the sterile field display may be a primary display and / or secondary display, allowing the surgeon to control one or more primary displays and / or secondary displays.

[0288] In one embodiment, a sterile field display may be employed to reconfigure wireless activation devices and their paired energy devices within the operating room (OR) when a surgeon hands over a device to another device. Figures 22A–22E show various types of sterile field control and data input consoles 6700, 6702, 6708, 6712, and 6714 according to various embodiments of the present disclosure. Each of the disclosed sterile field control and data input consoles 6700, 6702, 6708, 6712, and 6714 comprises at least one touchscreen 6701, 6704 / 6706, 6709, 6713, and 6716 input / output devices superimposed on the electronic visual display of an information processing system. The sterile field control and data input consoles 6700, 6702, 6708, 6712, and 6714 may include batteries as a power source. Some include a cable 6710 for connecting to a separate power source or for recharging the batteries. Users can provide input or control the information processing system via simple or multi-touch gestures by touching the touchscreens 6701, 6704 / 6706, 6709, 6713, and 6716 with a stylus, one or more fingers, or a surgical tool. Sterile field control and data entry consoles 6700, 6702, 6708, 6712, and 6714 may be used to reconfigure the wireless activation device and paired energy device in the operating room when a surgeon hands the device to another surgeon. For example, the sterile field display may be a primary display and / or secondary display, and it may be possible for the surgeon to control one or more primary displays and / or secondary displays.

[0289] The sterile field control and data entry consoles 6700, 6702, 6708, 6712, and 6714 may be used to receive advice from other operating rooms, and some or all of the operating room screens may be configured to mirror the other operating rooms so that the surgeon can see what is needed to assist. The sterile field control and data entry consoles 6700, 6702, 6708, 6712, and 6714 are configured to communicate with the surgical hub 206. Accordingly, the description of the surgical hub 206 described in relation to Figures 1 to 11 is incorporated into this section by reference.

[0290] Figure 22A shows a single-zone sterile field control and data input console 6700 according to one aspect of the present disclosure. The single-zone console 6700 is configured for use within a single zone in a sterile field. The single-zone console 6700 may have a secondary display. When deployed in a sterile field, the single-zone console 6700 can receive touchscreen input from a user in the sterile field. The touchscreen 6701 allows the user to interact directly with what is displayed, rather than using a mouse, touchpad, or other such device (other than a stylus or surgical tool). The single-zone console 6700 includes wireless communication circuitry for wirelessly communicating with a surgical hub 206. The single-zone console 6700 may allow the user to control a primary display and / or another secondary display.

[0291] Figure 22B shows a multizone sterile field control and data input console 6702 according to one aspect of the present disclosure. The multizone console 6702 comprises a first touchscreen 6704 for receiving input from a first zone of the sterile field and a second touchscreen 6706 for receiving input from a second zone of the sterile field. The multizone console 6702 may be a secondary display. The multizone console 6702 is configured to receive input from multiple users in the sterile field. The multizone console 6702 includes a wireless communication circuit for wirelessly communicating with the surgical hub 206. Thus, the multizone sterile field control and data input console 6702 comprises an interactive touchscreen display having multiple input and output zones. The multizone console 6702 may allow a user to control the primary display and / or another secondary display.

[0292] Figure 22C shows a tethered sterile field control and data input console 6708 according to one aspect of the present disclosure. The tethered console 6708 includes a cable 6710 for connecting the tethered console 6708 to the surgical hub 206 via a wired connection. The cable 6710 enables the tethered console 6708 to communicate via a wired link in addition to a wireless link. The cable 6710 also enables the tethered console 6708 to connect to a power source for powering the console 6708 and / or recharging the battery in the console 6708. The tethered console 6708 may be a secondary display. The tethered console 6708 may enable the user to control the primary display and / or another secondary display.

[0293] Figure 22D shows a battery-powered sterile field control and data input console 6712 according to one aspect of the present disclosure. The sterile field console 6712 is battery-powered and includes a wireless communication circuit for wirelessly communicating with the surgical hub 206. In one aspect, the sterile field console 6712 may be configured to communicate with any of the modules coupled to the hub 206, such as a generator module 240. Through the sterile field console 6712, the surgeon may adjust the power output level of the generator using a touchscreen 6713 interface. An example is described below in relation to Figure 22E. The sterile field console 6712 may be a secondary display. The sterile field console 6712 may allow the user to control the primary display and / or another secondary display.

[0294] Figure 22E shows a battery-powered sterile field control and data input console 6714 according to one aspect of the present disclosure. The sterile field console 6714 may include a user interface displayed on the generator's touchscreen. The surgeon may thus control the generator's output by touching up / down arrow icons 6718A, 6718B to increase / decrease the power output of the generator module 240. Additional icons 6719, among other features, allow access to the generator module settings 6174, volume 6178 directly from the sterile field console 6714 using + / - icons. The sterile field console 6714 may also be used to adjust or reconfigure the settings of other wireless activators or modules connected to the hub 206 in the operating room, and their paired energy devices, when the surgeon hands the sterile field console 6714 to another. The sterile field console 6714 may be a secondary display. The sterile field console 6714 may allow the user to control the primary display and / or another secondary display.

[0295] Figures 23A and 23B show a sterile field console 6700 used in a sterile field during a surgical procedure according to one aspect of the present disclosure. Figure 23 shows a sterile field console 6714 positioned in a sterile field near two surgeons performing surgery. Figure 23 shows one of the surgeons tapping the touchscreen 6701 of the sterile field console with a surgical tool 6722 to adjust the output of a modular device coupled to a surgical hub 206 and to reconfigure the modular device, or an energy device paired with a modular device coupled to a surgical hub 206.

[0296] The sterile field display may be employed as an interactive, scalable secondary display that allows the surgeon to overlay other feeds or images, such as a laser Doppler scanning array. In one embodiment, the sterile field display may be employed to retrieve preoperative scans or images for further examination. Once the vascular pathway and depth, as well as the device trajectory, are estimated, the surgeon employs the sterile field interactive, scalable secondary display that allows the surgeon to overlay other feeds or images.

[0297] Figure 24 is a schematic diagram 6770 illustrating techniques for estimating vascular pathways, depths, and instrument trajectories. Using a standard approach, before incising vessels 6772 and 6774 located beneath the surface of tissue 6775, the surgeon estimates the pathways and depths of vessels 6772 and 6774, as well as the trajectory 6776 that surgical instruments 6778 will take to reach vessels 6772 and 6774. Estimating the pathways and depths 6776 of vessels 6772 and 6774 located beneath the surface of tissue 6775 is often difficult because surgeons cannot accurately visualize the location of the pathways and depths 6776 of vessels 6772 and 6774.

[0298] Figures 25A–25D show multiple real-time images of virtual anatomical details for incision, including oblique views (Figures 25A, 25C) and lateral views (Figures 25B, 25D). Images may be displayed on a primary and / or secondary display. For example, images may be displayed on a tablet computer or a sterile field display of a sterile field control and data entry console, which is employed as an interactive, scalable secondary display that allows the surgeon to overlay other feeds or images, according to one aspect of the present disclosure. Images of the virtual anatomy may allow the surgeon to more accurately predict the pathways and depths of blood vessels 6772, 6774 located beneath the surface of tissue 6775, as shown in Figure 24, and the best trajectory 6776 of surgical instruments 6778.

[0299] Figure 25A is a perspective view of the virtual dissection 6780 displayed on a tablet computer or a secondary device such as a sterile field control and data entry console. Figure 25B is a side view of the virtual dissection 6780 shown in Figure 25A, according to one aspect of the present disclosure. Referring to Figures 25A to 25B, in one aspect, a surgeon uses a smart surgical device 6778 and a tablet computer to visualize the virtual dissection 6780 in real time in multiple views. The smart surgical device 6778 may include a display, which may be a secondary display. The tablet computer may include a display, which may be a primary display and / or a secondary display. The three-dimensional perspective view includes a portion of tissue 6775 in which blood vessels 6772, 6774 are located below the surface. The tissue portion is overlaid with a grid 6786 to enable the surgeon to visualize the scale and measure the pathway and depth of blood vessels 6772, 6774 at target locations 6782, 6784, which are marked by X respectively. The grid 6786 also assists the surgeon in determining the best trajectory 6776 for the surgical instrument 6778. As shown in the figure, the blood vessels 6772 and 6774 have abnormal vascular pathways.

[0300] Figure 25C shows a perspective view of a virtual anatomy 6780 for incision according to one aspect of the present disclosure. Figure 25D is a side view of the virtual anatomy 6780 for incision according to one aspect of the present disclosure. Referring to Figures 25C to 25D, a surgeon can use a tablet computer to zoom and pan 360-∞ to obtain the optimal view of the virtual anatomy 6780 for incision. The surgeon then determines the best route or trajectory 6776 for inserting the surgical instrument 6778 (e.g., an incision instrument in this example). The surgeon can view the anatomical structure in a three-dimensional perspective view or in any one of the six views. See, for example, the side view of the virtual anatomical structure and the insertion of the surgical instrument 6778 (e.g., an incision instrument) in Figure 25D.

[0301] In another embodiment, the sterile field control and data entry console may enable live chat between different departments, such as the oncology or pathology departments, to discuss margins or other details associated with imaging. The sterile field control and data entry console may enable the pathology department to communicate margin relationships within specimens to surgeons and display them to surgeons in real time using the sterile field console.

[0302] In another embodiment, the sterile field control and data entry console may be used to change the focus and field of view of its own image, or to control one of the other monitors coupled to the surgical hub. For example, the sterile field control and data entry console may be a primary display and / or secondary display, which may be used to control another primary display and / or secondary display.

[0303] In another embodiment, a sterile field control and data entry console may be used to display the status of any of the instruments or modules coupled to the surgical hub 206. Knowledge of which instruments coupled to the surgical hub 206 are in use may be obtained through information such as the instrument not being on an instrument pad or on-device sensor. Based on this information, the sterile field control and data entry console may change the display, change the configuration, switch power to drive one instrument and not another, and switch one code from capital to an instrument pad and multiple codes from there. Instrument diagnostics can be obtained to know that an instrument is inactive or not in use. Instrument diagnostics may be based on information such as the instrument not being on an instrument pad or on sensors on the instrument.

[0304] In another embodiment, the sterile field control and data entry console may be used as a learning tool. The console may display checklists, procedure steps, and / or a series of steps. A timer / clock may be displayed to measure the time required to complete the steps and / or procedures. The console may display room sound pressure levels as an indicator of activity, stress, etc.

[0305] Figures 26A to 26E show a touchscreen display 6890 that may be used in a sterile field according to one aspect of the present disclosure. The touchscreen display 6890 may be a primary display and / or a secondary display. Using the touchscreen display 6890, a surgeon may manipulate images 6892 displayed on the touchscreen display 6890 using a variety of gestures, such as, for example, drag-and-drop, scroll, zoom, rotate, tap, double-tap, flick, drag, swipe, pinch open, pinch close, touch-and-hold, and two-finger scrolling. Using the touchscreen display 6890, a surgeon may manipulate images 6892 that may be displayed on another primary display and / or a secondary display using a variety of gestures, such as, for example, drag-and-drop, scroll, zoom, rotate, tap, double-tap, flick, drag, swipe, pinch open, pinch close, touch-and-hold, and two-finger scrolling. The surgeon may also use gestures, such as gestures on the touchscreen display 6890, to move images or data displayed on the touchscreen display 6890 to another primary display and / or secondary display.

[0306] Figure 26A shows an image 6892 of a surgical site displayed on the touchscreen display 6890 in portrait mode. Figure 26B shows the touchscreen display 6890 rotated in landscape mode (e.g., arrow 6894), and the surgeon uses their index finger 6896 to scroll the image 6892 in the direction of the arrow. Figure 26C shows the surgeon using their index finger 6896 and thumb 6898 to pinch open and zoom out the image 6892 in the direction of arrow 6899. Figure 26D shows the surgeon using their index finger 6896 and thumb 6898 to pinch open and zoom out the image 6892 in the direction of arrow 6897. Figure 26E shows the touchscreen display 6890 rotated in two directions indicated by arrows 6894, 6896 so that the surgeon can view the image 6892 in different orientations.

[0307] Outside the sterile field, control and static displays are used that may differ from those used inside the sterile field. Control and static displays located outside the sterile field provide interactive and static displays for operating room (OR) and equipment control. Control and static displays located outside the sterile field may be primary and / or secondary displays. These may include secondary static displays for input and output, and secondary displays such as secondary touchscreens.

[0308] Non-sterile displays 107, 109, 119 (Figure 2) may be used outside the sterile field and may include monitors positioned on the walls of the operating room, on rotating stands, or on capital equipment. The displays may present feeds from the control devices to which they are mounted, and may display what is presented thereto.

[0309] One or more secondary displays, which may be secondary touch input screens located outside the sterile field, may be part of the visualization system 108 (Figure 2), part of the surgical hub 106 (Figure 2), or fixed-position touch monitors on a wall or rotating stand. The difference between a touch input screen and a static display may be that the user can interact with the touch input screen by changing what can be displayed on that particular monitor or other. In the case of capital equipment applications, it may be an interface for controlling the settings of connected capital equipment. Primary and / or secondary displays outside the sterile field may be used to preload surgeon preferences. For example, touch input screens and static displays outside the sterile field may be used to preload surgeon preferences (instrument settings and modes, lighting, procedures and preferred processes and sequences, music, etc.).

[0310] Secondary displays, such as secondary surgeon displays, may function similarly to sterile field input display devices but may include personal input displays with personal input devices that can be controlled by the surgeon. Secondary displays, such as personal secondary displays, may be implemented in many form factors, such as watches, miniature display pads, and interface glasses. Personal secondary displays may include control capabilities for display devices and may be placed on or controlled by the surgeon. Personal secondary displays may be keyed by the surgeon (e.g., specifically keyed by the surgeon) and may indicate this to one or more users, itself, one or more primary displays, one or more secondary displays, and / or other devices. Personal secondary displays can be used to grant permission to release devices. Personal secondary displays may be used to control one or more primary displays and / or secondary displays. For example, personal secondary displays may be used to control what is displayed on primary displays and / or secondary displays. As another example, personal secondary displays may be used to move data from one display to another.

[0311] A personal secondary display can be used to provide dedicated data to one of several surgeons who wish to monitor something that others do not want others to monitor. The personal secondary display may also be used as a command module. The personal secondary display may be held by the chief surgeon in the operating room and may give the surgeon control to override any other input from anyone else. The personal secondary display may be coupled to a short-range wireless (e.g., Bluetooth) microphone and / or earphone to allow the surgeon to have a separate conversation or call, or the personal secondary display may be used to broadcast to everyone else in the operating room or other departments. The surgeon may also use the microphone and / or earphone to issue verbal commands to the personal secondary display. The surgeon may also use gestures to provide one or more commands to the personal secondary display.

[0312] Figure 27 is a logic flow diagram 6920 of a process illustrating a control program or logic configuration for communication from inside a sterile field to an apparatus located outside the sterile field, according to one aspect of the present disclosure. In one aspect, the control unit may comprise an interactive touchscreen display, an interface configured to connect the interactive touchscreen display to a surgical hub, a processor, and memory connected to the processor. The memory may store instructions that can be executed by the processor to receive input commands from the interactive touchscreen display located inside the sterile field (6922), transmit the input commands to the surgical hub, and control an apparatus connected to the surgical hub located outside the sterile field (6924).

[0313] Figure 28 shows a second information layer overlaid on the first information layer. The second information layer includes a symbolic representation of the knife, overlapping with the detected location of the knife within the disposable loading unit (DLU) shown in the first information layer. Further examples are disclosed in U.S. Patent No. 9,283,054, “SURGICAL APPARATUS WITH INDICATOR,” issued March 15, 2016, which is incorporated herein by reference in its entirety.

[0314] Referring to Figure 28, the second information layer 6963 can overlay at least a portion of the first information layer 6962 on the display 6960. Furthermore, a touchscreen 6961, which may be a primary and / or secondary display, may enable the user to interact with the second information layer 6963 in relation to video feedback in the underlying first information layer 6962 on the display 6960. For example, the user can interact with the touchscreen 6961 to select, manipulate, reformat, resize, and / or otherwise modify the information displayed on the second information layer 6963. In one embodiment, the user can move the first and / or second information layer to one or more displays, which may include a primary and / or secondary display. In one embodiment, the user can interact with the touchscreen 6961 to interact with the second information layer 6963 in relation to a surgical instrument 6964 shown in the first information layer 6962 on the display 6960. The user may, for example, select a menu, category, and / or classification on the control panel 6967, and the second information layer 6963 and / or the control panel 6967 may be adjusted to reflect the user's selection. In various embodiments, the user may select a category from instrument feedback categories 6969 corresponding to a specific feature(s) of the surgical instrument 6964 shown in the first information layer 6962. The feedback corresponding to the category selected by the user may be moved, positioned, and / or "snapped" to a location on the display 6960 relating to a specific feature(s) of the surgical instrument 6964. For example, the selected feedback may be moved to a position close to and / or overlapping with one or more specific feature(s) of the surgical instrument 6964 shown in the first information layer 6962.

[0315] The instrument feedback menu 6969 may include multiple feedback categories and may relate to feedback data measured and / or detected by the surgical instrument 6964 during a surgical procedure. As described herein, the surgical instrument 6964 may detect and / or measure, for example, the position 6970 of the movable jaw between open and closed orientations, the thickness 6973 of the clamped tissue, the clamping force 6976 on the clamped tissue, the joint flexion 6974 of the DLU 6965, and / or the position 6971, velocity 6972, and / or force 6975 of the firing element. Furthermore, a feedback controller communicating with the surgical instrument 6964 may provide the sensed feedback to the display 6960, which can display the feedback in a second information layer 6963. As described herein, the selection, arrangement, and / or form of the feedback data displayed in the second information layer 6963 can be changed, for example, based on user input to the touchscreen 6961.

[0316] When the knife of DLU6965 is obscured from view by the end effector jaw 6966 and / or tissue T, for example, the operator may track and / or estimate the position of the knife in DLU6965 based on the changing value of the feedback data and / or the shift position of the feedback data relative to DLU6965 shown in the lower first information layer 6962.

[0317] In various embodiments, the display menu 6977 of the control panel 6967 may relate to multiple categories, such as unit systems 6978 and / or data modes 6979. In certain embodiments, the user can select unit system category 6978 to switch between unit systems, for example, between metric units and US customary units. In addition, the user can select data mode category 6979 to switch, for example, the type of numerical representation of feedback data and / or the type of graphical representation of feedback data. The numerical representation of feedback data can be displayed, for example, as a number and / or a percentage. Furthermore, the graphical representation of feedback data can be displayed, for example, as a function of time and / or distance. As described herein, the user can select the instrument controller menu 6980 from the control panel 6967 to input instructions for the surgical instrument 6964, which can be implemented, for example, via an instrument controller and / or microcontroller. The user can minimize or collapse the control panel 6967 by selecting the minimize / maximize icon 6968, and maximize or expand the control panel 6967 by re-selecting the minimize / maximize icon 6968.

[0318] Figure 29 shows a perspective view of a surgeon using a surgical instrument, including a handle assembly housing and a wireless circuit board, during a surgical procedure, with the surgeon wearing safety glasses. The wireless circuit board transmits signals to the safety glasses worn by the surgeon using the surgical instrument during the procedure. The signals are received by a wireless port on the safety glasses. One or more illuminators on the front lenses of the safety glasses change color, fade, or illuminate in response to the received signals to indicate information about the status of the surgical instrument to the surgeon. The illuminators can be positioned on the periphery of the front lenses so as not to obstruct the surgeon's line of sight. Further examples are disclosed in U.S. Patent No. 9,011,427, “SURGICAL INSTRUMENT WITH SAFETY GLASSES,” issued April 21, 2015, which is incorporated herein by reference in its entirety.

[0319] Figure 29 shows a version of safety glasses 6991 that may be worn by a surgeon 6992 during a surgical procedure while using a medical device. The safety glasses 6991 may be a primary display and / or a secondary display. The safety glasses 6991 may be used to determine the direction in which the surgeon 6992 is looking. For example, the safety glasses 6991 may analyze the movement of the surgeon's pupils (e.g., using an internal or external camera) to determine that the surgeon is looking at the monitor 6997. As another example, the safety glasses 6991 may use one or more sensors to track the movement of the surgeon's head to determine where the surgeon is looking (e.g., whether the surgeon is looking at the monitor 6997).

[0320] During use, a wireless communication board housed within the surgical instrument 6993 can communicate with a wireless port 6994 on the safety glasses 6991. The exemplary surgical instrument 6993 is battery-powered, but the instrument 6993 may be powered by cable or other means. The instrument 6993 includes an end effector. In particular, the wireless communication board (6995) transmits one or more wireless signals, indicated by arrows (B, C), to the wireless port (6994) of the safety glasses (6991). The safety glasses 6991 receive the signals, analyze the received signals, and display the indicated status information received by the signals on the lenses 6996 to the user, such as a surgeon 6992 wearing the safety glasses 6991.

[0321] The wireless communication board 6995 can transmit wireless signals to the surgical monitor 6997 so that the surgical monitor 6997 can display the indicated status information it has received to the surgeon 6992, as described herein. The surgical monitor 6997 may be a primary display and / or a secondary display.

[0322] Versions of the safety glasses 6991 may include an illumination device on the periphery of the safety glasses 6991. The illumination device provides peripheral vision sensory feedback of the device 6993, thereby enabling the safety glasses 6991 to communicate with the user wearing the safety glasses 6991. The illumination device may be, for example, a light-emitted diode ("LED"), a series of LEDs, or any other suitable illumination device known to those skilled in the art and evident from the teachings herein.

[0323] The LEDs may be positioned on the edge or side of the front lens of the safety glasses 6991 so as not to deviate from the center of the user's field of view, while remaining within the user's field of view, so as not to deviate from the surgical site to view the illumination device. The displayed light may flash and / or change color to communicate various aspects of information read from the instrument 6993 (e.g., system status information or tissue sensing information (i.e., whether the end effector has sufficiently cut and sealed the tissue)) to the wearer of the safety glasses 6991. Feedback from the housed wireless communication board 6995 may activate, flash, or change color the illumination device to indicate to the user information regarding the use of the instrument 6993. For example, the device may incorporate a feedback mechanism based on one or more sensed tissue parameters. In this case, a change in the device output based on this feedback, synchronized with a change in state, can send a signal to the safety glasses 6991 via the wireless communication board 6995 to trigger the activation of the illumination device. Such described means of activating the lighting device should not be considered limiting, as other means are intended to show the user status information of the instrument 6993 via the safety glasses 6991. Furthermore, the safety glasses 6991 may be disposable or reusable eyewear. A button-type power supply device, such as a button cell battery, may be used to power the wireless receiver and the LEDs of the modifications of the safety glasses (6991), which may also include a housed wireless board and three-color LEDs. Such a button-type power supply device can provide a low-cost means of providing sensory feedback of information about the instrument (6993) to the surgeon (6992) wearing the safety glasses (6991) when in use.

[0324] It is an unfortunate reality that the outcomes of all surgical procedures are not always optimal and / or successful. For example, if a failure event is detected and / or identified, a communication method can be used to separate surgical data that may be associated with the failure event (e.g., failure event surgical data) from surgical data that may not be associated with the failure event (e.g., non-failure event surgical data), and the surgical data that may be associated with the failure event (e.g., failure event data) can be communicated from the surgical hub 206 to the cloud-based system 205 on a priority basis for analysis. According to one aspect of this disclosure, failure event surgical data can be communicated from the surgical hub 206 to the cloud-based system 205 with priority over non-failure event surgical data.

[0325] Figure 30 illustrates various aspects of a system implementation method for identifying surgical data associated with failure events (e.g., failure event surgical data) and communicating the identified surgical data to a cloud-based system 205 on a priority basis. The method includes receiving surgical data in a surgical hub 206 (3838), the surgical data being associated with a surgical procedure, timestamping the surgical data (3840), identifying a failure event associated with the surgical procedure (3842), determining which of the surgical data is associated with the failure event (e.g., failure event surgical data) (3844), separating the surgical data associated with the failure event from all other surgical data received in the surgical hub 206 (e.g., non-failure event surgical data) (3846), chronologically series the surgical data associated with the failure event (3848), encrypting the surgical data associated with the failure event (3850), and communicating the encrypted surgical data to a cloud-based system 205 based on priority (3852).

[0326] More specifically, various surgical data may be captured during a surgical procedure, and the captured surgical data, as well as other surgical data associated with the surgical procedure, may be communicated to the surgical hub 206. Surgical data may include, for example, data associated with surgical devices / instruments used during surgery (e.g., Figure 5, surgical devices / instruments 235), data associated with the patient, data associated with the facility where the surgical procedure was performed, and data associated with the surgeon. Either before or after the surgical data is communicated to and received by the surgical hub 206, the surgical data may be time-stamped and all information that could identify a particular surgery, patient, or surgeon removed, thereby essentially anonymizing the information for further processing and analysis by the cloud-based system 205.

[0327] When a fault event is detected and / or identified (for example, either during or after a surgical procedure), the surgical hub 206 may determine which of the surgical data are associated with the fault event (e.g., fault event surgical data) and which of the surgical data are not associated with the surgical event (e.g., non-fault event surgical data). According to one aspect of the present disclosure, a fault event may include, for example, the detection of one or more misfired staples during a stapling portion of a surgical procedure. For example, in one aspect, referring to Figure 5, an endoscope 239 may take a snapshot while a surgical device / instrument 235, which includes an end effector containing a staple cartridge, is performing a stapling portion of a surgical procedure. In such an aspect, an imaging module 238 may compare the snapshot with stored images and / or images downloaded from a cloud-based system 205 that accurately convey the fired staples in order to detect misfired staples and / or evidence (leaks) of misfired staples. In another embodiment, the imaging module 238 may analyze the snapshot itself to detect misfired staples and / or evidence of misfired staples. In one alternative embodiment, the surgical hub 206 may communicate the snapshot to a cloud-based system 205, and components of the cloud-based system 205 may perform the various imaging module functions described above to detect misfired staples and / or evidence of misfired staples and report the detection to the surgical hub 206. According to another embodiment of the present disclosure, the failure event may include detection of a tissue temperature below the expected temperature in the tissue sealing portion of the surgical procedure, and / or visual indication of excessive bleeding or exudation after the surgical procedure (e.g., via endoscope 239, Figure 5). For example, in one embodiment, referring to Figure 5, the surgical device / instrument 235 may comprise an end effector including a temperature sensor and a surgical hub 206, and / or a cloud-based system may detect an inappropriate / low sealing temperature by comparing at least one temperature detected by the temperature sensor (e.g., between tissue sealing portions of a surgical procedure) with an expected and / or stored temperature and / or temperature range associated with the surgical procedure. In another embodiment, the endoscope 239 may take snapshots during the surgical procedure.In one such embodiment, the imaging module 238 may compare snapshots with stored images and / or images downloaded from the cloud-based system 205 that convey tissue that has been accurately sealed at the expected temperature, in order to detect evidence of an inappropriate / insufficient sealing temperature (e.g., charring, exudation / bleeding). Furthermore, in one such embodiment, the imaging module 238 may analyze the snapshots themselves to detect evidence of an inappropriate / insufficient sealing temperature (e.g., charring, exudation / bleeding). As another example, the surgical hub 206 may communicate the snapshots to the cloud-based system 205, and components of the cloud-based system 205 may perform the various imaging module functions described above to detect evidence of an inappropriate / insufficient sealing temperature and report the detection to the surgical hub 206. According to various embodiments described herein, in response to a detected and / or identified fault event, the surgical hub 206 may download a program from the cloud-based system 205 (for example, a program to modify surgical device / instrument parameters to prevent misfired staples, a program to modify surgical device / instrument parameters to ensure accurate sealing temperatures) for execution by the surgical device / instrument 235 to correct the detected problem.

[0328] In some embodiments, a failure event may be considered to encompass a specific period, and one or more (e.g., all) surgical data associated with that period may be considered to be associated with the failure event.

[0329] After surgical data related to a failure event has been identified, the identified surgical data (e.g., failure event surgical data) may be separated or isolated from some or all of the other surgical data related to the surgical procedure (e.g., non-failure event surgical data). This separation can be achieved, for example, by tagging or flagging the identified surgical data to store it separately from all other surgical data associated with the surgical procedure, or by storing only the other surgical data while continuing to process the identified surgical data for subsequent priority communication to the cloud-based system 205. According to various embodiments, tagging or flagging of the identified surgical data may occur during the communication process when the datagram is generated, as will be described in more detail below.

[0330] Timestamps of surgical data (for example, either before or after the surgical data is received in the surgical hub) may be used by components of the surgical hub 206 to chronologically sequence the identified surgical data associated with a failure event. Components of the surgical hub 206 that utilize timestamps to chronologically sequence the identified surgical data may be, for example, the processor module 232, the processor 244 of the computer system 210, and / or a combination thereof. By chronologically sequenced the identified surgical data, the cloud-based system 205 and / or other parties can subsequently better understand the conditions leading to the occurrence of a failure event and possibly pinpoint the exact cause of the failure event, thereby providing knowledge to potentially mitigate similar failure events occurring during similar surgical procedures performed in the future.

[0331] If the identified surgical data is time-series, the time-series surgical data may be encrypted in a manner similar to that described above for the encryption of generator data. Thus, the identified surgical data may be encrypted to help ensure the confidentiality of the identified surgical data, either while it is stored in the surgical hub 206 or while it is being transmitted to the cloud-based system 205 using the Internet or other computer network. In various embodiments, components of the surgical hub 206 utilize an encryption algorithm to convert the identified surgical data from a readable version to an encoded version, thereby forming encrypted surgical data associated with a failure event. Components of the surgical hub that utilize an encryption algorithm may be, for example, a processor module 232, a processor 244 of the computer system 210, and / or a combination thereof. The encryption algorithm used may be a symmetric or asymmetric encryption algorithm.

[0332] After the identified surgical data is encrypted, components of the surgical hub can communicate the encrypted surgical data associated with a failure event (e.g., encrypted failure event surgical data) to a cloud-based system 205. Components of the surgical hub that communicate the encrypted surgical data to the cloud-based system 205 may be, for example, a processor module 232, a hub / switch 207 / 209 of the modular communication hub 203, a router 211 of the modular communication hub 203, or a communication module 247 of the computer system 210. According to various embodiments, communication of encrypted surgical data (e.g., encrypted failure event surgical data) over the internet can provide a datagram containing the encrypted surgical data to be delivered and can follow IP, which can provide an addressing method used to label the datagram with source and destination information. The datagram may include fields containing flags or tags that identify the encrypted surgical data (e.g., encrypted failure event surgical data) as being prioritized over other unprioritized surgical data (e.g., encrypted non-failure event surgical data).

[0333] In some embodiments, once a failure event associated with a surgical procedure is identified, the surgical hub 206 and / or cloud-based system 205 may subsequently flag or tag the surgical devices / instruments 235 used during the surgical procedure for malfunction and / or removal. For example, in one embodiment, information associated with the surgical devices / instruments 235 and stored in the surgical hub 206 and / or cloud-based system 205 (e.g., serial number, ID) may be used to effectively prevent the surgical devices / instruments 235 from being used again (e.g., being blacklisted). In another embodiment, information associated with the surgical devices / instruments (e.g., serial number, ID) may initiate the printing of shipping labels and shipping instructions for returning the surgical devices / instruments 235 to the manufacturer or other designated party so that a thorough analysis / inspection of the surgical devices / instruments 235 can be carried out (e.g., to determine the cause of the failure). According to various embodiments described herein, once the cause of a failure is determined (for example, via the surgical hub 206 and / or the cloud-based system 205), the surgical hub 206 may download from the cloud-based system 205 a program for execution by the surgical device / instrument 235 (i.e., a program that modifies surgical device / instrument parameters to prevent the failure from occurring again) that corrects the determined cause of the failure.

[0334] In some embodiments, the primary and / or secondary displays may be used to provide or display notifications that an operational error has occurred. For example, when a failure event associated with a surgical procedure is identified, the surgical hub 206 and / or the cloud-based system 205 may transmit an error message that is displayed on one or more primary and / or secondary displays. The error message may indicate to the user that a failure event has occurred, may provide instructions to correct the error, may provide recommendations for correcting the error, or may provide instructions that may modify the surgical procedure. For example, an error message on the primary display may provide instructions for a surgical error that may have occurred to the patient as a result of the failure event. As another example, an error message on the secondary display may provide instructions to the user on how to remove a misfired staple and reload the staple cartridge.

[0335] In some embodiments, the surgical hub 206 and / or cloud-based system 205 may also provide / display reminders to administrators, staff, and / or other parties to physically remove the surgical device / instrument 235 from the operating room (e.g., if it is detected that it is still present in the operating room) and / or send the surgical device / instrument 235 to the manufacturer or other designated party (e.g., via the hub display 215 and / or surgical device / instrument display 237). In one embodiment, the reminders may be configured to be provided / displayed periodically until the administrator can remove the flag or tag of the surgical device / instrument 235 from the surgical hub 206 and / or cloud-based system 205. In various embodiments, the administrator may remove the flag or tag once the administrator can confirm that the surgical device / instrument 235 has been received by the manufacturer or other designated party (e.g., system tracking of the surgical device / instrument 235 via its serial number / ID). By using the methods described herein to flag and / or track surgical data associated with failure events, closed-loop control of surgical data associated with failure events and / or surgical devices / instruments 235 can be achieved. It will be understood that the surgical hub 206 can be used to effectively manage the use (or non-use) of surgical devices / instruments 235 that have been or may be used during surgical procedures.

[0336] In various aspects of this disclosure, the surgical hub 206 and / or cloud-based system 205 may want to control which components (e.g., surgical devices / instruments 235, energy devices 241) are used within its interactive surgical system 100 / 200 to perform surgical procedures (e.g., to avoid the use of unauthorized or knockoff components in order to minimize future failure events).

[0337] Therefore, in various aspects of this disclosure, the interactive surgical system 100 may comprise multiple surgical hubs 106, so that the cloud-based system 105 of the interactive surgical system 100 and / or each surgical hub 106 may wish to track the combination of components-surgical hubs used over time. In one aspect, when a component (see Figure 5, e.g., surgical device / instrument 235, energy device 241) is connected to / used with a particular surgical hub 106 (e.g., surgical device / instrument 235 is wired / wireless connected to a particular surgical hub 106, and energy device 241 is connected to a particular surgical hub 106 via a generator module 240), the particular surgical hub 106 may communicate a record / block of its connection / use to the cloud-based system 105 of the interactive surgical system 100 and / or other surgical hubs 106 (e.g., linking the unique identifiers of each connected device). For example, during / after the connection / use of the energy device 241, a particular surgical hub 106 may communicate records / blocks to the cloud-based system 105 and / or other surgical hubs 106 of the interactive surgical system 100 (e.g., linking the unique identifier of the energy device 241 to the unique identifier of the generator module 240 and to the unique identifier of the particular surgical hub 106). In one such embodiment, if this is the first time a component (e.g., an energy device) is connected to / used with the surgical hub 106 of the interactive surgical system 100, the cloud-based system 105 and / or each surgical hub 106 of the interactive surgical system 100 may store the records / blocks as generated records / blocks. In one such embodiment, the generated records / blocks stored in the cloud-based system 105 and / or each surgical hub 106 may include a timestamp.However, in one such embodiment, if this is not the first time a component (e.g., energy device 241) has connected to / used the...

Claims

1. A surgical hub for prioritizing data on a display, wherein the surgical hub is A processor is provided, and the processor is Based on contextual data, determine the first surgical operation in which medical instruments will be used during the medical procedure, To determine first display data associated with the first surgical procedure and related to the user performing the first surgical procedure using the medical instrument, Based on the first surgical procedure and the context data, a second surgical procedure using the medical instrument is determined. Based on user identification information and the context data, determine second display data associated with the second surgical procedure and relating to the user performing the second surgical procedure using the medical instrument; A surgical hub configured to send a message to the display instructing it to prioritize the second display data.

2. The surgical hub according to claim 1, wherein the processor is further configured to determine the medical procedure.

3. The surgical hub according to claim 1 or 2, wherein the second surgical operation is performed after the completion of the first surgical operation of the medical procedure.

4. The surgical hub according to any one of claims 1 to 3, wherein the message is a first message, and the processor is further configured to send a second message to the medical instrument to instruct the medical instrument to be configured according to the second surgical operation.

5. The surgical hub according to any one of claims 1 to 4, wherein the context data includes one or more of the following: data received from the medical device, the status of the medical device, the status of a subsystem of the medical device, the status of a component of the medical device, the status of the motor of the medical device, the end effector orientation, the reload status, the configuration of the medical device, and operational information.

6. The surgical hub according to any one of claims 1 to 5, wherein the processor is further configured to determine that the second surgical operation using the medical instrument is one or more of a critical task, a critical task, a dangerous task, or an error correction task.

7. A surgical hub for prioritizing data on a display, wherein the surgical hub is A processor is provided, and the processor is Based on contextual data, determine the first surgical operation in which medical instruments will be used during the medical procedure, To determine first display data associated with the first surgical procedure and related to the user performing the first surgical procedure using the medical instrument, Receiving instrument data from the medical instrument associated with the first surgical procedure, Based on the first surgical procedure, the instrument data, and the medical procedure, a second surgical procedure using the medical instrument is determined. Based on user identification information and the context data, determine second display data associated with the second surgical procedure and relating to the user performing the second surgical procedure using the medical instrument; A surgical hub configured to send a message to the display instructing it to prioritize the second display data.

8. The surgical hub according to claim 7, wherein the instrument data includes one or more of the following: user feedback, user-adjusted parameters for the medical device, standby time, firing force parameter (FTF), clamp compression parameter, indicator of cartridge loading, cartridge status, indicator of activated control, indicator of deactivated medical device control, battery power level, and status of the medical device.

9. The surgical hub according to claim 7 or 8, wherein the processor is further configured to determine an error by analyzing the instrument data from the medical instrument using the context data.

10. The surgical hub according to claim 9, wherein the displayed data indicates the error.

11. The surgical hub according to claim 9 or 10, wherein the processor is further configured to determine one or more instructions for resolving the error, and the second display data includes the one or more instructions for resolving the error.

12. The context data is the first context data, and the processor, Receiving the second context data, The surgical hub according to claim 7 or 8, further configured to determine an error that occurred during the medical procedure based on the second context data.

13. The surgical hub according to any one of claims 9 to 12, wherein the second surgical operation is a corrective surgical operation for correcting the error that occurred during the medical procedure, and the processor is further configured to determine one or more instructions for assisting the user in performing the corrective surgical operation, and the second display data includes the one or more instructions for assisting the user in performing the corrective surgical operation.

Images