Controlling surgical systems across surgical barriers

Abstract

The present invention discloses a surgical system assembly. The surgical system assembly includes a first surgical system and a second surgical system coupled to the first surgical system. The second surgical system includes a control circuit. The control circuit is configured to be able to operate in a first mode or a second mode, and to control one or more functions of the first surgical system when the second surgical system is in the second mode.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. non-provisional patent application serial number 16 / 182,232, filed on November 6, 2018, entitled “CONTROL OF A SURGICAL SYSTEM THROUGH A SURGICAL BARRIER,” the disclosure of which is incorporated herein by reference in its entirety.

[0003] This application claims priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 62 / 692,747, filed on September 10, 2018, entitled “INDIRECT COMMAND AND CONTROL OF A FIRST OPERATING ROOM SYSTEM THROUGH THE USE OF A SECOND OPERATING ROOM SYSTEM WITHIN A STERILE FIELD WHERE THE SECONDOPERATING ROOM SYSTEM HAS PRIMARY AND SECONDARY OPERATING MODES,” the disclosure of which is incorporated herein by reference in its entirety.

[0004] This application also claims priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 62 / 692,747, filed on June 30, 2018, entitled “SMART ACTIVATION OF AN ENERGY DEVICE BY ANOTHER DEVICE,” U.S. Provisional Patent Application No. 62 / 692,748, filed on June 30, 2018, entitled “SMART ENERGY ARCHITECTURE,” and U.S. Provisional Patent Application No. 62 / 692,768, filed on June 30, 2018, entitled “SMART ENERGY DEVICES,” the disclosure of each of which is incorporated herein by reference in its entirety.

[0005] This application claims priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application No. 62 / 659,900, filed on April 19, 2018, entitled “METHOD OF HUB COMMUNICATION,” the disclosure of which is incorporated herein by reference in its entirety.

[0006] This application also claims, pursuant to 35 U.S.C. 119(e), U.S. Provisional Patent Application No. 62 / 650,898, filed on March 30, 2018, entitled “CAPACITIVE COUPLED RETURN PATH PAD WITH SEPARABLE ARRAY ELEMENTS,” U.S. Provisional Patent Application Serial No. 62 / 650,887, filed on March 30, 2018, entitled “SURGICAL SYSTEMS WITH OPTIMIZED SENSING CAPABILITIES,” U.S. Provisional Patent Application Serial No. 62 / 650,882, filed on March 30, 2018, entitled “SMOKE EVACUATION MODULE FOR INTERACTIVE SURGICAL PLATFORM,” and U.S. Provisional Patent Application Serial No. 62 / 650,883, filed on March 30, 2018, entitled “SURGICAL SMOKE EVACUATION SENSING AND No. 62 / 650,877, the disclosures of each of which are incorporated herein by reference in their entirety.

[0007] This application also claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application Serial No. 62 / 640,417, filed on March 8, 2018, entitled “TEMPERATURE CONTROL IN ULTRASONIC DEVICE AND CONTROL SYSTEM THEREFOR,” and U.S. Provisional Patent Application Serial No. 62 / 640,415, filed on March 8, 2018, entitled “ESTIMATING STATEOF ULTRASONIC END EFFECTOR AND CONTROL SYSTEM THEREFOR,” the disclosure of each of which is incorporated herein by reference in its entirety.

[0008] This application also claims priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application Serial No. 62 / 611,341, filed on December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” U.S. Provisional Patent Application Serial No. 62 / 611,340, filed on December 28, 2017, entitled “CLOUD-BASED MEDICAL ANALYTICS,” and U.S. Provisional Patent Application Serial No. 62 / 611,339, filed on December 28, 2017, entitled “ROBOT ASSISTED SURGICAL PLATFORM,” the disclosure of each of which is incorporated herein by reference in its entirety. Background Art

[0009] The present disclosure relates to various surgical systems. Surgery is typically performed in operating theaters or rooms in a medical facility, such as, for example, a hospital. A sterile field is typically created around the patient. The sterile field may include appropriately dressed scrubbed team members, as well as all equipment and fixtures in the area. Various surgical devices and systems are utilized in performing surgical procedures. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The various aspects described herein (relating to surgical organizations and methods) and further objects and advantages thereof may best be understood by reference to the following description taken in conjunction with the following drawings.

[0011] Figure 1 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0012] Figure 2 A surgical system for performing a surgical procedure in an operating room according to at least one aspect of the present disclosure.

[0013] Figure 3 A surgical hub is paired with a visualization system, a robotic system, and a smart instrument according to at least one aspect of the present disclosure.

[0014] Figure 4 A partial perspective view of a surgical hub housing and a combination generator module slidably receivable in a drawer of the surgical hub housing according to at least one aspect of the present disclosure.

[0015] Figure 5 is a perspective view of a combination generator module having bipolar, ultrasonic and monopolar contacts and smoke evacuation components according to at least one aspect of the present disclosure.

[0016] Figure 6A single power bus attachment for multiple lateral docking ports of a lateral modular housing configured to receive multiple modules is shown in accordance with at least one aspect of the present disclosure.

[0017] Figure 7 A vertical modular housing configured to receive a plurality of modules according to at least one aspect of the present disclosure is shown.

[0018] Figure 8 A surgical data network including a modular communication hub configured to enable connecting modular devices located in one or more operating rooms of a medical facility or any room in a medical facility dedicated to surgical procedures to a cloud is shown in accordance with at least one aspect of the present disclosure.

[0019] Fig. 9 A computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0020] Fig.10 A surgical hub comprising multiple modules coupled to a modular control tower is shown in accordance with at least one aspect of the present disclosure.

[0021] Fig.11 An aspect of a universal serial bus (USB) hub device in accordance with at least one aspect of the present disclosure is shown.

[0022] Fig.12 is a block diagram of a cloud computing system according to at least one aspect of the present disclosure, the cloud computing system including a plurality of smart surgical instruments coupled to a surgical hub, which is connectable to a cloud component of the cloud computing system.

[0023] Fig.13 The present invention is a functional module architecture of a cloud computing system according to at least one aspect of the present disclosure.

[0024] Fig.14 A diagram of a situational awareness surgical system is shown in accordance with at least one aspect of the present disclosure.

[0025] Fig.15 A timeline depicting situational awareness of a surgical hub according to at least one aspect of the present disclosure.

[0026] Fig.16 0014] A diagram of a display screen for surgical procedures depicting a surgical site and a distal portion of a surgical device at the surgical site according to at least one aspect of the present disclosure.

[0027] Fig.17 According to at least one aspect of the present disclosure Fig.16 View of a surgical device extending through a surgical barrier into a surgical site.

[0028] Fig.18 According to at least one aspect of the present disclosure Fig.16 and Fig.17 A perspective view of a handle portion of a surgical device having an input switch for switching the surgical device between operating modes.

[0029] Fig.19 A diagram depicting a wearable device in communication with a surgical instrument to facilitate pairing and handover of the surgical instrument according to at least one aspect of the present disclosure.

[0030] Fig. 20 is a diagram of a wearable wrist device according to at least one aspect of the present disclosure.

[0031] Fig.21 is a diagram of a wearable ring-like device according to at least one aspect of the present disclosure.

[0032] Fig.22A 0026] A first view of a display screen configured to receive operator input to control a first surgical device, which is a combination energy device, according to at least one aspect of the present disclosure.

[0033] Fig. 22B According to at least one aspect of the present disclosure Fig.22A A second view of a display screen of the present invention, wherein the display screen is configured to receive operator input to control a second surgical device, which is a stapler. DETAILED DESCRIPTION

[0034] The applicant of the present application owns the following U.S. patent applications filed on November 6, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0035] U.S. Patent Application No. 16 / 182,224, entitled “SURGICAL NETWORK, INSTRUMENT, AND CLOUD RESPONSES BASED ON VALIDATION OF RECEIVED DATASET AND AUTHENTICATION OF ITS SOURCE AND INTEGRITY”;

[0036] U.S. Patent Application No. 16 / 182,230, entitled “SURGICAL SYSTEM FOR PRESENTING INFORMATION INTERPRETED FROMEXTERNAL DATA”;

[0037] U.S. Patent Application No. 16 / 182,233, entitled “MODIFICATION OF SURGICAL SYSTEMS CONTROL PROGRAMS BASED ON MACHINE LEARNING”;

[0038] U.S. Patent Application No. 16 / 182,239, entitled “ADJUSTMENT OF DEVICE CONTROL PROGRAMS BASED ON STRATIFIED CONTEXTUAL DATA IN ADDITION TO THE DATA”;

[0039] U.S. Patent Application No. 16 / 182,243, entitled “SURGICAL HUB AND MODULAR DEVICE RESPONSES ADJUSTMENT BASED ONSITUATIONAL AWARENESS”;

[0040] U.S. Patent Application No. 16 / 182,248, entitled “DETECTION AND ESCALATION OF SECURITY RESPONSES OF SURGICALINSTRUMENTS TO INCREASING SEVERITY THREATS”;

[0041] U.S. Patent Application No. 16 / 182,251, entitled “INTERACTIVE SURGICAL SYSTEM”;

[0042] U.S. Patent Application No. 16 / 182,260, entitled “AUTOMATED DATA SCALING, ALIGNMENT, AND ORGANIZING BASED ON PREDEFINED PARAMETERS WITH IN SURGICAL NETWORKS”;

[0043] U.S. Patent Application No. 16 / 182,267, entitled “SENSING THE PATIENT POSITION AND CONTACT UTILIZING THE MONO-POLAR RETURN PAD ELECTRODE TO PROVIDE SITUATIONAL AWARENESS TO A SURGICAL NETWORK”;

[0044] U.S. Patent Application No. 16 / 182,249, entitled “POWERED SURGICAL TOOL WITH PREDEFINED ADJUSTABLE CONTROLAL GORITHM FOR CONTROLLING END EFFECTOR PARAMETER”; U.S. Patent Application No. 16 / 182,246, entitled “ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES”;

[0045] U.S. Patent Application No. 16 / 182,256, entitled “ADJUSTMENT OF A SURGICAL DEVICE FUNCTION BASED ONSITUATIONAL AWARENESS”;

[0046] U.S. Patent Application No. 16 / 182,242, entitled “REAL-TIME ANALYSIS OF COMPREHENSIVE COST OF ALL INSTRUMENTATION USED IN SURGERY UTILIZING DATA FLUIDITY TO TRACK INSTRUMENTSTHROUGH STOCKING AND IN-HOUSE PROCESSES”;

[0047] U.S. Patent Application No. 16 / 182,255, entitled “USAGE AND TECHNIQUE ANALYSIS OF SURGEON / STAFF PERFORMANCE AGAINST A BASELINE TO OPTIMIZE DEVICE UTILIZATION AND PERFORMANCE FOR BOTH CURRENT AND FUTURE PROCEDURES”;

[0048] U.S. Patent Application No. 16 / 182,269, entitled “IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVEPLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE”;

[0049] U.S. Patent Application No. 16 / 182,278, entitled “COMMUNICATION OF DATA WHERE A SURGICAL NETWORK IS USING CONTEXT OF THE DATA AND REQUIREMENTS OF A RECEIVING SYSTEM / USER TO INFLUENCE INCLUSION OR LINKAGE OF DATA AND METADATA TO ESTABLISH CONTINUITY”;

[0050] U.S. Patent Application No. 16 / 182,290, entitled “SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION”;

[0051] U.S. Patent Application No. 16 / 182,227, entitled “SURGICAL NETWORK DETERMINATION OF PRIORITIZATION OF COMMUNICATION, INTERACTION, OR PROCESSING BASED ON SYSTEM OR DEVICE NEEDS”;

[0052] U.S. Patent Application No. 16 / 182,231, entitled “WIRELESS PAIRING OF A SURGICAL DEVICE WITH ANOTHER DEVICEWITHIN A STERILE SURGICAL FIELD BASED ON THE USAGE AND SITUATIONAL AWARENESS OF DEVICES”;

[0053] U.S. Patent Application No. 16 / 182,229, entitled “ADJUSTMENT OF STAPLE HEIGHT OF AT LEAST ONE ROW OF STAPLESBASED ON THE SENSED TISSUE THICKNESS OR FORCE IN CLOSING”;

[0054] U.S. Patent Application No. 16 / 182,234, entitled “STAPLING DEVICE WITH BOTH COMPULSORY AND DISCRETIONARY LOCKOUTS BASED ON SENSED PARAMETERS”;

[0055] U.S. Patent Application No. 16 / 182,240, entitled “POWERED STAPLING DEVICE CONFIGURED TO ADJUST FORCE, ADVANCEMENT SPEED, AND OVERALL STROKE OF CUTTING MEMBER BASED ON SENSED PARAMETER OF FIRING OR CLAMPING”;

[0056] U.S. Patent Application No. 16 / 182,235, entitled “VARIATION OF RADIO FREQUENCY AND ULTRASONIC POWER LEVEL INCOOPERATION WITH VARYING CLAMP ARM PRESSURE TO ACHIEVE PREDEFINED HEAT FLUXOR POWER APPLIED TO TISSUE”; and

[0057] • U.S. Patent Application No. 16 / 182,238, entitled “ULTRASONIC ENERGY DEVICE WHICH VARIES PRESSURE APPLIED BY CLAMP ARM TO PROVIDE THRESHOLD CONTROL PRESSURE AT A CUT PROGRESSION LOCATION.”

[0058] The applicant of the present application owns the following U.S. patent applications filed on September 10, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0059] U.S. Provisional Patent Application No. 62 / 729,183, entitled “A CONTROL FOR A SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE THAT ADJUSTS ITS FUNCTION BASED ON A SENSED SITUATION OR USAGE”;

[0060] U.S. Provisional Patent Application No. 62 / 729,177, entitled “AUTOMATED DATA SCALING, ALIGNMENT, AND ORGANIZING BASED ON PREDEFIINED PARAMETERS WITHIN A SURGICAL NETWORK BEFORE TRANSMISSION”;

[0061] U.S. Provisional Patent Application No. 62 / 729,176, entitled “INDIRECT COMMAND AND CONTROL OF A FIRST OPERATING ROOM SYSTEM THROUGH THE USE OF A SECOND OPERATING ROOM SYSTEM WITHIN A STERILEFIELD WHERE THE SECOND OPERATING ROOM SYSTEM HAS PRIMARY AND SECONDARY OPERATING MODES”;

[0062] U.S. Provisional Patent Application No. 62 / 729,185, entitled “POWERED STAPLING DEVICE THAT IS CAPABLE OF ADJUSTING FORCE, ADVANCEMENT SPEED, AND OVERALL STROKE OF CUTTING MEMBER OF THE DEVICE BASED ONSENSED PARAMETER OF FIRING OR CLAMPING”;

[0063] U.S. Provisional Patent Application No. 62 / 729,184, entitled “POWERED SURGICAL TOOL WITH A PREDEFINED ADJUSTABLE CONTROL ALGORITHM FOR CONTROLLING AT LEAST ONE END EFFECTOR PARAMETER AND A MEANS FOR LIMITING THE ADJUSTMENT”;

[0064] U.S. Provisional Patent Application No. 62 / 729,182, entitled “SENSING THE PATIENT POSITION AND CONTACT UTILIZING THE MONOPOLAR RETURN PAD ELECTRODE TO PROVIDE SITUATIONAL AWARENESS TO THE HUB”;

[0065] U.S. Provisional Patent Application No. 62 / 729,191, entitled “SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION”;

[0066] U.S. Provisional Patent Application No. 62 / 729,195, entitled “ULTRASONIC ENERGY DEVICE WHICH VARIES PRESSURE APPLIED BY CLAMP ARM TO PROVIDE THRESHOLD CONTROL PRESSURE AT A CUT PROGRESSION LOCATION”; and

[0067] U.S. Provisional Patent Application No. 62 / 729,186, entitled “WIRELESS PAIRING OF A SURGICAL DEVICE WITH ANOTHER DEVICEWITHIN A STERILE SURGICAL FIELD BASED ON THE USAGE AND SITUATIONAL AWARENESS OF DEVICES”

[0068] The applicant of the present application owns the following U.S. patent applications filed on August 28, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0069] U.S. Patent Application No. 16 / 115,214, entitled “ESTIMATING STATE OF ULTRASONIC END EFFECTOR AND CONTROL SYSTEM THEREFOR”;

[0070] U.S. Patent Application No. 16 / 115,205, entitled “TEMPERATURE CONTROL OF ULTRASONIC END EFFECTOR AND CONTROL SYSTEM THEREFOR”;

[0071] U.S. Patent Application 16 / 115,233, entitled “RADIO FREQUENCY ENERGY DEVICE FOR DELIVERING COMBINEDELECTRICAL SIGNALS”;

[0072] U.S. Patent Application No. 16 / 115,208, entitled “CONTROLLING AN ULTRASONIC SURGICAL INSTRUMENT ACCORDING TOTISSUE LOCATION”;

[0073] U.S. Patent Application No. 16 / 115,220, entitled “CONTROLLING ACTIVATION OF AN ULTRASONIC SURGICAL INSTRUMENT ACCORDING TO THE PRESENCE OF TISSUE”;

[0074] U.S. Patent Application No. 16 / 115,232, entitled “DETERMINING TISSUE COMPOSITION VIA AN ULTRASONIC SYSTEM”;

[0075] U.S. Patent Application No. 16 / 115,239, entitled “DETERMINING THE STATE OF AN ULTRASONIC ELECTROMECHANICAL SYSTEM ACCORDING TO FREQUENCY SHIFT”;

[0076] U.S. Patent Application No. 16 / 115,247, entitled “DETERMINING THE STATE OF AN ULTRASONIC END EFFECTOR”;

[0077] U.S. Patent Application No. 16 / 115,211, entitled “SITUATIONAL AWARENESS OF ELECTROSURGICAL SYSTEMS”;

[0078] U.S. Patent Application No. 16 / 115,226, entitled “MECHANISMS FOR CONTROLLING DIFFERENT ELECTROMECHANICAL SYSTEMS OF AN ELECTROSURGICAL INSTRUMENT”;

[0079] U.S. Patent Application No. 16 / 115,240, entitled “DETECTION OF END EFFECTOR IMMERSION IN LIQUID”;

[0080] U.S. Patent Application No. 16 / 115,249, entitled “INTERRUPTION OF ENERGY DUE TO INADVERTENT CAPACITIVE COUPLING”;

[0081] U.S. Patent Application No. 16 / 115,256, entitled “INCREASING RADIO FREQUENCY TO CREATE PAD-LESS MONOPOLAR LOOP”;

[0082] U.S. Patent Application No. 16 / 115,223, entitled “BIPOLAR COMBINATION DEVICE THAT AUTOMATICALLY ADJUSTSPRESSURE BASED ON ENERGY MODALITY”; and

[0083] • U.S. Patent Application No. 16 / 115,238, entitled “ACTIVATION OF ENERGY DEVICES.”

[0084] The applicant of the present application owns the following U.S. patent applications filed on August 23, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0085] U.S. Provisional Patent Application No. 62 / 721,995, entitled “CONTROLLING AN ULTRASONIC SURGICAL INSTRUMENT ACCORDING TOTISSUE LOCATION”;

[0086] U.S. Provisional Patent Application No. 62 / 721,998, entitled “SITUATIONAL AWARENESS OF ELECTROSURGICAL SYSTEMS”;

[0087] U.S. Provisional Patent Application No. 62 / 721,999, entitled “INTERRUPTION OF ENERGY DUE TO INADVERTENT CAPACITIVE COUPLING”;

[0088] U.S. Provisional Patent Application No. 62 / 721,994, entitled “BIPOLAR COMBINATION DEVICE THAT AUTOMATICALLY ADJUSTSPRESSURE BASED ON ENERGY MODALITY”; and

[0089] U.S. Provisional Patent Application No. 62 / 721,996, entitled “RADIO FREQUENCY ENERGY DEVICE FOR DELIVERING COMBINEDELECTRICAL SIGNALS”

[0090] The applicant of the present application owns the following U.S. patent applications filed on June 30, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0091] U.S. Provisional Patent Application 62 / 692,747, entitled “SMART ACTIVATION OF AN ENERGY DEVICE BY ANOTHER DEVICE”;

[0092] U.S. Provisional Patent Application No. 62 / 692,748, entitled “SMART ENERGY ARCHITURE”; and

[0093] • U.S. Provisional Patent Application 62 / 692,768, entitled “SMART ENERGY DEVICES.”

[0094] The applicant of the present application owns the following U.S. patent applications filed on June 29, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0095] U.S. Patent Application Serial No. 16 / 024,090, entitled “CAPACITIVE COUPLED RETURN PATH PAD WITH SEPARABLE ARRAYELEMENTS”;

[0096] U.S. Patent Application Serial No. 16 / 024,057, entitled “CONTROLLING A SURGICAL INSTRUMENT ACCORDING TO SENSED CLOSURE PARAMETERS”;

[0097] U.S. Patent Application Serial No. 16 / 024,067, entitled “SYSTEMS FOR ADJUSTING END EFFECTOR PARAMETERS BASED ON PERIOPERATIVE INFORMATION”;

[0098] U.S. Patent Application Serial No. 16 / 024,075, entitled “SAFETY SYSTEMS FOR SMART POWERED SURGICAL STAPLING”;

[0099] U.S. Patent Application Serial No. 16 / 024,083, entitled “SAFETY SYSTEMS FOR SMART POWERED SURGICAL STAPLING”;

[0100] U.S. Patent Application Serial No. 16 / 024,094, entitled “SURGICAL SYSTEMS FOR DETECTING END EFFECTOR TISSUEDISTRIBUTION IRREGULARITIES”;

[0101] U.S. Patent Application Serial No. 16 / 024,138, entitled “SYSTEMS FOR DETECTING PROXIMITY OF SURGICAL END EFFECTOR TOCANCEROUS TISSUE”;

[0102] U.S. Patent Application Serial No. 16 / 024,150, entitled “SURGICAL INSTRUMENT CARTRIDGE SENSOR ASSEMBLIES”;

[0103] U.S. Patent Application Serial No. 16 / 024,160, entitled “VARIABLE OUTPUT CARTRIDGE SENSOR ASSEMBLY”;

[0104] U.S. Patent Application Serial No. 16 / 024,124, entitled “SURGICAL INSTRUMENT HAVING A FLEXIBLE ELECTRODE”;

[0105] U.S. Patent Application Serial No. 16 / 024,132, entitled “SURGICAL INSTRUMENT HAVING A FLEXIBLE CIRCUIT”;

[0106] U.S. Patent Application Serial No. 16 / 024,141, entitled “SURGICAL INSTRUMENT WITH A TISSUE MARKING ASSEMBLY”;

[0107] U.S. Patent Application Serial No. 16 / 024,162, entitled “SURGICAL SYSTEMS WITH PRIORITIZED DATA TRANSMISSION CAPABILITIES”;

[0108] U.S. Patent Application Serial No. 16 / 024,066, entitled “SURGICAL EVACUATION SENSING AND MOTOR CONTROL”;

[0109] U.S. Patent Application Serial No. 16 / 024,096, entitled “SURGICAL EVACUATION SENSOR ARRANGEMENTS”;

[0110] U.S. Patent Application Serial No. 16 / 024,116, entitled “SURGICAL EVACUATION FLOW PATHS”;

[0111] U.S. Patent Application Serial No. 16 / 024,149, entitled “SURGICAL EVACUATION SENSING AND GENERATOR CONTROL”;

[0112] U.S. Patent Application Serial No. 16 / 024,180, entitled “SURGICAL EVACUATION SENSING AND DISPLAY”;

[0113] U.S. Patent Application Serial No. 16 / 024,245, entitled “COMMUNICATION OF SMOKE EVACUATION SYSTEM PARAMETERS TO HUBOR CLOUD IN SMOKE EVACUATION MODULE FOR INTERACTIVE SURGICAL PLATFORM”;

[0114] U.S. Patent Application Serial No. 16 / 024,258, entitled “SMOKE EVACUATION SYSTEM INCLUDING A SEGMENTED CONTROLCIRCUIT FOR INTERACTIVE SURGICAL PLATFORM”;

[0115] U.S. Patent Application Serial No. 16 / 024,265, entitled “SURGICAL EVACUATION SYSTEM WITH A COMMUNICATION CIRCUIT FOR COMMUNICATION BETWEEN A FILTER AND A SMOKE EVACUATION DEVICE”; and

[0116] • U.S. Patent Application Serial No. 16 / 024,273, entitled "DUAL IN-SERIES LARGE AND SMALL DROPLET FILTERS."

[0117] The applicant of the present application owns the following U.S. provisional patent applications filed on June 28, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0118] U.S. Provisional Patent Application Serial No. 62 / 691,228, entitled “A METHOD OF USING REINFORCED FLEX CIRCUITS WITH MULTIPLESENSORS WITH ELECTROSURGICAL DEVICES”;

[0119] U.S. Provisional Patent Application Serial No. 62 / 691,227, entitled “CONTROLLING A SURGICAL INSTRUMENT ACCORDING TO SENSED CLOSURE PARAMETERS”;

[0120] U.S. Provisional Patent Application Serial No. 62 / 691,230, entitled “SURGICAL INSTRUMENT HAVING A FLEXIBLE ELECTRODE”;

[0121] U.S. Provisional Patent Application Serial No. 62 / 691,219, entitled “SURGICAL EVACUATION SENSING AND MOTOR CONTROL”;

[0122] U.S. Provisional Patent Application Serial No. 62 / 691,257, entitled “COMMUNICATION OF SMOKE EVACUATION SYSTEM PARAMETERS TO HUBOR CLOUD IN SMOKE EVACUATION MODULE FOR INTERACTIVE SURGICAL PLATFORM”;

[0123] U.S. Provisional Patent Application Serial No. 62 / 691,262, entitled “SURGICAL EVACUATION SYSTEM WITH A COMMUNICATION CIRCUIT FOR COMMUNICATION BETWEEN A FILTER AND A SMOKE EVACUATION DEVICE”; and

[0124] • U.S. Provisional Patent Application Serial No. 62 / 691,251, entitled “DUAL IN-SERIES LARGE AND SMALL DROPLET FILTERS.”

[0125] The applicant of the present application owns the following U.S. provisional patent applications filed on April 19, 2018, the disclosures of which are incorporated herein by reference in their entirety:

[0126] • U.S. Provisional Patent Application Serial No. 62 / 659,900, entitled “METHOD OF HUB COMMUNICATION.”

[0127] The applicant of the present application owns the following U.S. provisional patent applications filed on March 30, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0128] U.S. Provisional Patent Application No. 62 / 650,898, filed on March 30, 2018, entitled “CAPACITIVE COUPLED RETURN PATH PAD WITH SEPARABLE ARRAY ELEMENTS”;

[0129] U.S. Provisional Patent Application Serial No. 62 / 650,887, entitled “SURGICAL SYSTEMS WITH OPTIMIZED SENSING CAPABILITIES”;

[0130] U.S. Patent Application Serial No. 62 / 650,882, entitled “SMOKE EVACUATION MODULE FOR INTERACTIVE SURGICAL PLATFORM”; and

[0131] • U.S. Patent Application Serial No. 62 / 650,877, entitled “SURGICAL SMOKE EVACUATION SENSING AND CONTROLS.”

[0132] The applicant of the present application owns the following U.S. patent applications filed on March 29, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0133] U.S. Patent Application Serial No. 15 / 940,641, entitled “INTERACTIVE SURGICAL SYSTEMS WITH ENCRYPTED COMMUNICATION CAPABILITIES”;

[0134] U.S. Patent Application Serial No. 15 / 940,648, entitled “INTERACTIVE SURGICAL SYSTEMS WITH CONDITION HANDLING OF DEVICES AND DATA CAPABILITIES”;

[0135] U.S. Patent Application Serial No. 15 / 940,656, entitled “SURGICAL HUB COORDINATION OF CONTROL AND COMMUNICATION OF OPERATING ROOM DEVICES”;

[0136] U.S. Patent Application Serial No. 15 / 940,666, entitled “SPATIAL AWARENESS OF SURGICAL HUBS IN OPERATING ROOMS”;

[0137] U.S. Patent Application Serial No. 15 / 940,670, entitled “COOPERATIVE UTILIZATION OF DATA DERIVED FROM SECONDARY SOURCES BY INTELLIGENT SURGICAL HUBS”;

[0138] U.S. Patent Application Serial No. 15 / 940,677, entitled “SURGICAL HUB CONTROL ARRANGEMENTS”;

[0139] U.S. Patent Application Serial No. 15 / 940,632, entitled “DATA STRIPPING METHOD TO INTERROGATE PATIENT RECORDS AND CREATE ANONYMIZED RECORD”;

[0140] U.S. Patent Application Serial No. 15 / 940,640, entitled “COMMUNICATION HUB AND STORAGE DEVICE FOR STORING PARAMETERS AND STATUS OF A SURGICAL DEVICE TO BE SHARED WITH CLOUD BASED ANALYTICS SYSTEMS”;

[0141] U.S. Patent Application Serial No. 15 / 940,645, entitled “SELF DESCRIBING DATA PACKETS GENERATED AT AN ISSUING INSTRUMENT”;

[0142] U.S. Patent Application Serial No. 15 / 940,649, entitled “DATA PAIRING TO INTERCONNECT A DEVICE MEASURED PARAMETER WITH AN OUTCOME”;

[0143] U.S. Patent Application Serial No. 15 / 940,654, entitled “SURGICAL HUB SITUATIONAL AWARENESS”;

[0144] U.S. Patent Application Serial No. 15 / 940,663, entitled “SURGICAL SYSTEM DISTRIBUTED PROCESSING”;

[0145] U.S. Patent Application Serial No. 15 / 940,668, entitled “AGGREGATION AND REPORTING OF SURGICAL HUB DATA”;

[0146] U.S. Patent Application Serial No. 15 / 940,671, entitled “SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES INOPERATING THEATER”;

[0147] U.S. Patent Application Serial No. 15 / 940,686, entitled “DISPLAY OF ALIGNMENT OF STAPLE CARTRIDGE TO PRIOR LINEAR STAPLE LINE”;

[0148] U.S. Patent Application Serial No. 15 / 940,700, entitled “STERILE FIELD INTERACTIVE CONTROL DISPLAYS”;

[0149] U.S. Patent Application Serial No. 15 / 940,629, entitled “COMPUTER IMPLEMENTED INTERACTIVE SURGICAL SYSTEMS”;

[0150] U.S. Patent Application Serial No. 15 / 940,704, entitled “USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT”;

[0151] U.S. Patent Application Serial No. 15 / 940,722, entitled “CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY”;

[0152] • U.S. Patent Application Serial No. 15 / 940,742, entitled “DUAL CMOS ARRAY IMAGING”.

[0153] U.S. Patent Application Serial No. 15 / 940,636, entitled “ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL DEVICES”;

[0154] U.S. Patent Application Serial No. 15 / 940,653, entitled “ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL HUBS”;

[0155] U.S. Patent Application Serial No. 15 / 940,660, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR CUSTOMIZATION AND RECOMMENDATIONS TO A USER”;

[0156] U.S. Patent Application Serial No. 15 / 940,679, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR LINKING OF LOCAL USAGETRENDS WITH THE RESOURCE ACQUISITION BEHAVIORS OF LARGER DATA SET”;

[0157] U.S. Patent Application Serial No. 15 / 940,694, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR MEDICAL FACILITY SEGMENTED INDIVIDUALIZATION OF INSTRUMENT FUNCTION”;

[0158] U.S. Patent Application Serial No. 15 / 940,634, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR SECURITY AND AUTHENTICATION TRENDS AND REACTIVE MEASURES”;

[0159] U.S. Patent Application Serial No. 15 / 940,706, entitled “DATA HANDLING AND PRIORITIZATION IN A CLOUD ANALYTICS NETWORK”;

[0160] U.S. Patent Application Serial No. 15 / 940,675, entitled “CLOUD INTERFACE FOR COUPLED SURGICAL DEVICES”;

[0161] U.S. Patent Application Serial No. 15 / 940,627, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0162] U.S. Patent Application Serial No. 15 / 940,637, entitled “COMMUNICATION ARRANGEMENTS FOR ROBOT-ASSISTED SURGICALPLATFORMS”;

[0163] U.S. Patent Application Serial No. 15 / 940,642, entitled “CONTROLS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0164] U.S. Patent Application Serial No. 15 / 940,676, entitled “AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICALPLATFORMS”;

[0165] U.S. Patent Application Serial No. 15 / 940,680, entitled “CONTROLLERS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0166] U.S. Patent Application Serial No. 15 / 940,683, entitled “COOPERATIVE SURGICAL ACTIONS FOR ROBOT-ASSISTED SURGICALPLATFORMS”;

[0167] U.S. Patent Application Serial No. 15 / 940,690, entitled “DISPLAY ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0168] as well as

[0169] • U.S. Patent Application Serial No. 15 / 940,711, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS.”

[0170] The applicant of the present application owns the following U.S. provisional patent applications filed on March 28, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0171] U.S. Provisional Patent Application Serial No. 62 / 649,302, entitled “INTERACTIVE SURGICAL SYSTEMS WITH ENCRYPTED COMMUNICATION CAPABILITIES”;

[0172] U.S. Provisional Patent Application Serial No. 62 / 649,294, entitled “DATA STRIPPING METHOD TO INTERROGATE PATIENT RECORDS AND CREATE ANONYMIZED RECORD”;

[0173] U.S. Provisional Patent Application Serial No. 62 / 649,300, entitled “SURGICAL HUB SITUATIONAL AWARENESS”;

[0174] U.S. Provisional Patent Application Serial No. 62 / 649,309, entitled “SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES INOPERATING THEATER”;

[0175] U.S. Provisional Patent Application Serial No. 62 / 649,310, entitled “COMPUTER IMPLEMENTED INTERACTIVE SURGICAL SYSTEMS”;

[0176] U.S. Provisional Patent Application Serial No. 62 / 649,291, entitled “USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT”;

[0177] U.S. Provisional Patent Application Serial No. 62 / 649,296, entitled “ADAPTIVE CONTROL PROGRAM UPDATES FOR SURGICAL DEVICES”;

[0178] U.S. Provisional Patent Application Serial No. 62 / 649,333, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR CUSTOMIZATION AND RECOMMENDATIONS TO A USER”;

[0179] U.S. Provisional Patent Application Serial No. 62 / 649,327, entitled “CLOUD-BASED MEDICAL ANALYTICS FOR SECURITY AND AUTHENTICATION TRENDS AND REACTIVE MEASURES”;

[0180] U.S. Provisional Patent Application Serial No. 62 / 649,315, entitled “DATA HANDLING AND PRIORITIZATION IN A CLOUD ANALYTICS NETWORK”;

[0181] U.S. Provisional Patent Application Serial No. 62 / 649,313, entitled “CLOUD INTERFACE FOR COUPLED SURGICAL DEVICES”;

[0182] U.S. Provisional Patent Application Serial No. 62 / 649,320, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS”;

[0183] U.S. Provisional Patent Application Serial No. 62 / 649,307, entitled “AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICALPLATFORMS”; and

[0184] • U.S. Provisional Patent Application Serial No. 62 / 649,323, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS.”

[0185] The applicant of the present application owns the following U.S. provisional patent applications filed on March 8, 2018, the disclosures of each of which are incorporated herein by reference in their entirety:

[0186] U.S. Provisional Patent Application Serial No. 62 / 640,417, entitled “TEMPERATURE CONTROL IN ULTRASONIC DEVICE AND CONTROL SYSTEMTHEREFOR”; and

[0187] • U.S. Provisional Patent Application Serial No. 62 / 640,415, entitled "ESTIMATING STATE OF ULTRASONIC END EFFECTOR AND CONTROL SYSTEM THEREFOR."

[0188] The applicant of the present application owns the following U.S. provisional patent applications filed on December 28, 2017, the disclosures of each of which are incorporated herein by reference in their entirety:

[0189] U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM”;

[0190] U.S. Provisional Patent Application Serial No. 62 / 611,340, entitled “CLOUD-BASED MEDICAL ANALYTICS”; and

[0191] • U.S. Patent Application Serial No. 62 / 611,339, entitled "ROBOT ASSISTED SURGICAL PLATFORM."

[0192] Before describing in detail various aspects of the surgical device and generator, it should be noted that the application or use of the illustrative examples is not limited to the details of the configuration and arrangement of the components shown in the drawings and detailed description. The illustrative examples may be implemented alone or in combination with other aspects, variations and modifications, and may be practiced or performed in a variety of ways. In addition, unless otherwise indicated, the terms and expressions used herein are selected for the convenience of the reader to describe the exemplary embodiments and are not for limiting purposes. Moreover, it should be understood that one or more of the aspects, expressions of aspects and / or examples described below may be combined with any one or more of the other aspects, expressions of aspects and / or examples described below.

[0193] Surgical Hub

[0194] See also Figure 1 , the computer-implemented interactive surgical system 100 includes one or more surgical systems 102 and a cloud-based system (e.g., a cloud 104 that may include a remote server 113 coupled to a storage device 105). Each surgical system 102 includes at least one surgical hub 106 in communication with the cloud 104 that may include the remote server 113. In one example, as Figure 1 As shown in , the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with the hub 106. In some aspects, the surgical system 102 may include M number of hubs 106, N number of visualization systems 108, O number of robotic systems 110, and P number of handheld intelligent surgical instruments 112, where M, N, O, and P are integers greater than or equal to one.

[0195] In all respects, reference Figures 1 to 7 The smart device 112 described herein may be implemented as a surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B). Smart device 112 (eg, device 1 a to 1 n ) such as surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) is configured to be able to operate in the surgical data network 201, as referenced Figure 8 described.

[0196] Figure 2 An example of a surgical system 102 is shown for performing a surgical procedure on a patient lying flat on an operating table 114 in a surgical operating room 116. A robotic system 110 is used as part of the surgical system 102 during a surgical procedure. The robotic system 110 includes a surgeon's console 118, a patient side cart 120 (surgical robot), and a surgical robotic hub 122. The patient side cart 120 can manipulate at least one removably coupled surgical tool 117 through a minimally invasive incision in the patient's body while the surgeon observes the surgical site through the surgeon's console 118. Images of the surgical site can be obtained through a medical imaging device 124, which can be manipulated by the patient side cart 120 to orient the imaging device 124. The robotic hub 122 can be used to process images of the surgical site for subsequent display to the surgeon through the surgeon's console 118.

[0197] Other types of robotic systems may be readily adapted for use with the surgical system 102. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in U.S. Provisional Patent Application Serial No. 62 / 611,339, filed on December 28, 2017, entitled “ROBOT ASSISTED SURGICAL PLATFORM,” the disclosure of which is incorporated herein by reference in its entirety.

[0198] Various examples of cloud-based analytics performed by cloud 104 and suitable for use in the present disclosure are described in U.S. Provisional Patent Application Serial No. 62 / 611,340, filed on December 28, 2017, entitled “CLOUD-BASED MEDICAL ANALYTICS,” the disclosure of which is incorporated herein by reference in its entirety.

[0199] In various aspects, imaging device 124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0200] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0201] One or more illumination sources may be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescence spectrum) is that portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and may be referred to as visible light or simply light. A typical human eye will respond to wavelengths in the range of about 380 nm to about 750 nm in air.

[0202] The invisible spectrum (i.e., non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0203] In various aspects, the imaging device 124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophago-duodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0204] In one aspect, the imaging device employs multispectral monitoring to discern topography and underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths may be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible range, such as IR and ultraviolet. Spectral imaging may allow for the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title “Advanced Imaging Acquisition Module” in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring may be a useful tool for repositioning the surgical field after completing the surgical task to perform one or more of the previously described tests on the treated tissue.

[0205] It is self-evident that during any surgery, the operating room and surgical equipment need to be strictly sterilized. The strict sanitation and sterilization conditions required in the "surgical room" (i.e., operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including imaging device 124 and its attachments and components. It should be understood that the sterile field can be considered as a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered as the area around the patient prepared for surgery. The sterile field can include properly dressed scrubbed team members, as well as all equipment and fixtures in the area.

[0206] In various aspects, the visualization system 108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically arranged relative to the sterile field, such as Figure 2 . In one aspect, the visualization system 108 includes interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described in U.S. Provisional Patent Application Serial No. 62 / 611,341, filed on December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” under the title “Advanced Imaging Acquisition Module,” the disclosure of which is incorporated herein by reference in its entirety.

[0207] like Figure 2 , the main display 119 is positioned in the sterile field to be visible to the operator at the operating table 114. In addition, the visualization tower 111 is positioned outside the sterile field. The visualization tower 111 includes a first non-sterile display 107 and a second non-sterile display 109 that are separated from each other. The visualization system 108 guided by the hub 106 is configured to be able to coordinate the flow of information to the operators inside and outside the sterile field using the displays 107, 109 and 119. For example, the hub 106 can enable the imaging system 108 to display a snapshot of the surgical site recorded by the imaging device 124 on the non-sterile display 107 or 109, while maintaining real-time feedback of the surgical site on the main display 119. The snapshot on the non-sterile display 107 or 109 can allow the non-sterile operator to perform diagnostic steps related to the surgical procedure, for example.

[0208] In one aspect, the hub 106 is further configured to be able to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 111 to the primary display 119 within the sterile field, where it can be viewed by a sterile operator at the operating table. In one example, the input can be a modified form of a snapshot displayed on the non-sterile display 107 or 109, which can be routed to the primary display 119 via the hub 106.

[0209] See also Figure 2 , surgical instrument 112 is used in a surgical procedure as part of surgical system 102. Hub 106 is further configured to coordinate information flow to a display of surgical instrument 112. For example, coordinate information flow is further described in U.S. Provisional Patent Application Serial No. 62 / 611,341, filed on December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator at visualization tower 111 can be routed by hub 106 to surgical instrument display 115 within the sterile field, where the operator of surgical instrument 112 can observe the input or feedback. Exemplary surgical instruments suitable for use with surgical system 102 are described in U.S. provisional patent application serial number 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, under the title “Surgical Instrument Hardware,” the disclosure of which is incorporated herein by reference in its entirety.

[0210] See now Figure 3 , the hub 106 is depicted as communicating with the visualization system 108, the robotic system 110, and the handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140 (which may include a monopolar generator 142, a bipolar generator 144, and / or an ultrasonic generator 143), a communication module 130, a processor module 132, and a storage array 134. In certain aspects, as Figure 3 As shown in , the hub 106 also includes a smoke evacuation module 126 , a suction / rinsing module 128 and / or an OR mapping module 133 .

[0211] During surgery, energy application to tissue for sealing and / or cutting is usually associated with smoke evacuation, suction of excess fluid and / or flushing of tissue. Fluid lines, power lines and / or data lines from different sources are usually tangled during surgery. Valuable time can be lost in solving this problem during surgery. Disconnecting a line may require disconnecting the line from its corresponding module, which may require resetting the module. The hub modular housing 136 provides a unified environment for managing power lines, data lines and fluid lines, which reduces the frequency of tangling between such lines.

[0212] Aspects of the present disclosure provide a surgical hub for a surgical procedure involving the application of energy to tissue at a surgical site. The surgical hub includes a hub housing and a combination generator module that can be slidably received in a docking base of the hub housing. The docking base includes data contacts and power contacts. The combination generator module includes two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, and a monopolar RF energy generator component seated in a single unit. In one aspect, the combination generator module also includes a smoke evacuation device for connecting the combination generator module to at least one energy delivery cable of a surgical instrument, at least one smoke evacuation device configured to be able to evacuate smoke, fluids, and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke evacuation device.

[0213] In one aspect, the fluid line is a first fluid line and the second fluid line extends from the remote surgical site to an aspiration and irrigation module slidably received in the hub housing. In one aspect, the hub housing includes a fluid interface.

[0214] Certain surgical procedures may require more than one energy type to be applied to tissue. One energy type may be more conducive to cutting tissue, while another different energy type may be more conducive to sealing tissue. For example, a bipolar generator may be used to seal tissue, while an ultrasonic generator may be used to cut sealed tissue. Aspects of the present disclosure provide a solution in which the hub modular housing 136 is configured to accommodate different generators and facilitate interactive communication between them. One of the advantages of the hub modular housing 136 is the ability to quickly remove and / or replace various modules.

[0215] Aspects of the present disclosure provide a modular surgical housing for use in a surgical procedure involving application of energy to tissue. The modular surgical housing includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking base including a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the first energy generator module is slidably movable out of electrical engagement with the first power and data contacts,

[0216] Further describing the above, the modular surgical shell also includes a second energy generator module, which is configured to generate a second energy different from the first energy for application to tissue, and a second docking base, which includes a second docking port, the second docking port including second data and power contacts, wherein the second energy generator module is slidably movable into electrical engagement with the power and data contacts, and wherein the second energy generator is slidably movable out of electrical contact with the second power and data contacts.

[0217] In addition, the modular surgical shell also includes a communication bus between the first docking port and the second docking port, which is configured to facilitate communication between the first energy generator module and the second energy generator module.

[0218] See also Figure 3-7 , aspects of the present disclosure are presented as a hub modular housing 136 that allows modular integration of the generator module 140, the smoke evacuation module 126, and the suction / irrigation module 128. The hub modular housing 136 also facilitates interactive communication between the modules 140, 126, 128. Figure 5 As shown in , the generator module 140 may be a generator module having integrated monopolar devices, bipolar devices, and ultrasonic devices supported in a single housing unit 139 that is slidably inserted into the hub modular housing 136. Figure 5 As shown in , the generator module 140 may be configured to be connectable to a monopolar device 146, a bipolar device 147, and an ultrasonic device 148. Alternatively, the generator module 140 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 136. The hub modular housing 136 may be configured to facilitate the insertion of multiple generators and interactive communication between the generators docked into the hub modular housing 136 so that the generators will act as a single generator.

[0219] In one aspect, the hub modular housing 136 includes a modular power and communications backplane 149 with external and wireless communications connectors to enable removable attachment of the modules 140, 126, 128 and interactive communications therebetween.

[0220] In one aspect, the hub modular housing 136 includes a docking base or drawer 151 (also referred to herein as a drawer) that is configured to slidably receive the modules 140, 126, 128. Figure 4 A partial perspective view of a surgical hub housing 136 and a combination generator module 145 that can be slidably received in a docking station 151 of the surgical hub housing 136 is shown. A docking port 152 having power and data contacts on the back of the combination generator module 145 is configured to engage the corresponding docking port 150 with the power and data contacts of the corresponding docking station 151 of the hub modular housing 136 when the combination generator module 145 is slid into place within the corresponding docking station 151 of the hub module housing 136. In one aspect, the combination generator module 145 includes bipolar, ultrasonic and monopolar modules and a smoke evacuation module integrated together into a single housing unit 139, such as Figure 5 as shown in .

[0221] In various aspects, the smoke evacuation module 126 includes a fluid line 154 that delivers captured / collected smoke and / or fluid from the surgical site to, for example, the smoke evacuation module 126. Vacuum suction from the smoke evacuation module 126 can draw smoke into an opening of a common conduit at the surgical site. The common conduit coupled to the fluid line can be in the form of a flexible tube that terminates at the smoke evacuation module 126. The common conduit and the fluid line define a fluid path that extends toward the smoke evacuation module 126 received in the hub housing 136.

[0222] In various aspects, the suction / irrigation module 128 is coupled to a surgical tool including an aspiration fluid line and a suction fluid line. In one example, the aspiration fluid line and the suction fluid line are in the form of flexible tubes extending from the surgical site toward the suction / irrigation module 128. One or more drive systems can be configured to be able to flush fluid to the surgical site and to aspirate fluid from the surgical site.

[0223] In one aspect, the surgical tool comprises a shaft having an end effector at its distal end and at least one energy treatment associated with the end effector, a suction tube, and a flushing tube. The suction tube may have an inlet at its distal end, and the suction tube extends through the shaft. Similarly, the suction tube may extend through the shaft and may have an inlet adjacent to an energy delivery tool. The energy delivery tool is configured to be able to deliver ultrasonic energy and / or RF energy to a surgical site, and is coupled to a generator module 140 by a cable initially extending through the shaft.

[0224] The irrigation tube may be in fluid communication with a fluid source, and the aspiration tube may be in fluid communication with a vacuum source. The fluid source and / or vacuum source may be seated in the suction / irrigation module 128. In one example, the fluid source and / or vacuum source may be seated in the hub housing 136 independently of the suction / irrigation module 128. In such examples, the fluid interface may be configured to connect the suction / irrigation module 128 to the fluid source and / or vacuum source.

[0225] In one aspect, the modules 140, 126, 128 and / or their corresponding docking stations on the hub modular housing 136 may include alignment features configured to align the docking ports of the modules to engage with their corresponding ports in the docking stations of the hub modular housing 136. For example, Figure 4 As shown in , the combination generator module 145 includes a side bracket 155, which is configured to be slidably engaged with a corresponding bracket 156 of a corresponding docking base 151 of the hub modular housing 136. The bracket cooperates to guide the docking port contacts of the combination generator module 145 to electrically engage with the docking port contacts of the hub modular housing 136.

[0226] In some aspects, the drawers 151 of the hub modular housing 136 are the same or substantially the same size, and the size of the module is adjusted to be received in the drawer 151. For example, the side brackets 155 and / or 156 can be larger or smaller depending on the size of the module. In other aspects, the drawers 151 are different sizes and each is designed to accommodate a specific module.

[0227] Additionally, the contacts of a particular module may be keyed to engage the contacts of a particular drawer to avoid inserting a module into a drawer with mismatched contacts.

[0228] like Figure 4 As shown in , the docking port 150 of one drawer 151 can be coupled to the docking port 150 of another drawer 151 via a communication link 157 to facilitate interactive communication between modules seated in the hub modular housing 136. Alternatively or additionally, the docking port 150 of the hub modular housing 136 can facilitate wireless interactive communication between modules seated in the hub modular housing 136. Any suitable wireless communication can be employed, such as, for example, Air Titan-Bluetooth.

[0229] Figure 6A single power bus attachment is shown for multiple lateral docking ports of a lateral modular housing 160 that is configured to receive multiple modules of a surgical hub 206. The lateral modular housing 160 is configured to receive and interconnect modules 161 laterally. The modules 161 are slidably inserted into docking bases 162 of the lateral modular housing 160 that includes a bottom plate for interconnecting the modules 161. Figure 6 As shown in FIG, the modules 161 are arranged transversely in the transverse modular housing 160. Alternatively, the modules 161 may be arranged vertically in the transverse modular housing.

[0230] Figure 7 A vertical modular housing 164 is shown that is configured to receive a plurality of modules 165 of a surgical hub 106. The modules 165 are slidably inserted into a docking station or drawer 167 of the vertical modular housing 164, which includes a bottom plate for interconnecting the modules 165. Although the drawers 167 of the vertical modular housing 164 are arranged vertically, in some cases, the vertical modular housing 164 may include drawers arranged horizontally. In addition, the modules 165 can interact with each other through the docking ports of the vertical modular housing 164. Figure 7 In the example of , a display 177 is provided for displaying data related to the operation of the module 165. In addition, the vertical modular housing 164 includes a main module 178 that seats a plurality of sub-modules that are slidably received in the main module 178.

[0231] In various aspects, the imaging module 138 includes an integrated video processor and a modular light source, and is suitable for use with various imaging devices. In one aspect, the imaging device is composed of a modular housing that can be equipped with a light source module and a camera module. The housing can be a disposable housing. In at least one example, the disposable housing is removably coupled to a reusable controller, light source module, and camera module. The light source module and / or camera module can be selectively selected depending on the type of surgical procedure. In one aspect, the camera module includes a CCD sensor. On the other hand, the camera module includes a CMOS sensor. On the other hand, the camera module is configured for scanning beam imaging. Similarly, the light source module can be configured to be able to deliver white light or different light, depending on the surgical procedure.

[0232] During a surgical procedure, it may be inefficient to remove a surgical device from the surgical field and replace the surgical device with another surgical device that includes a different camera or a different light source. Temporarily losing sight of the surgical field may lead to undesirable consequences. The modular imaging device of the present disclosure is configured to allow for mid-stream replacement of a light source module or a camera module during a surgical procedure without having to remove the imaging device from the surgical field.

[0233] In one aspect, an imaging device includes a tubular housing including a plurality of channels. A first channel is configured to slidably receive a camera module, which can be configured to snap-fit ​​into engagement with the first channel. A second channel is configured to slidably receive a light source module, which can be configured to snap-fit ​​into engagement with the second channel. In another example, the camera module and / or the light source module can be rotated into a final position within their respective channels. A threaded engagement can be used in place of a snap-fit ​​engagement.

[0234] In various examples, multiple imaging devices are placed at different locations in the surgical field to provide multiple views. The imaging module 138 can be configured to be able to switch between imaging devices to provide the best view. In various aspects, the imaging module 138 can be configured to be able to integrate images from different imaging devices.

[0235] Various image processors and imaging devices suitable for use with the present disclosure are described in U.S. Pat. No. 7,995,045, issued on Aug. 9, 2011, entitled “COMBINED SBI AND CONVENTIONAL IMAGE PROCESSOR”, which is incorporated herein by reference in its entirety. In addition, U.S. Pat. No. 7,982,776, issued on Jul. 19, 2011, entitled “SBI MOTION ARTIFACTREMOVAL APPARATUS AND METHOD”, describes various systems for removing motion artifacts from image data, which is incorporated herein by reference in its entirety. Such systems may be integrated with imaging module 138. In addition, U.S. Patent Application Publication No. 2011 / 0306840, published on December 15, 2011, entitled “CONTROLLABLE MAGNETIC SOURCE TO FIXTUREINTRACORPOREAL APPARATUS,” and U.S. Patent Application Publication No. 2014 / 0243597, published on August 28, 2014, entitled “SYSTEM FOR PERFORMING A MINIMALLY INVASIVE SURGICAL PROCEDURE,” each of which is incorporated herein by reference in its entirety.

[0236] Figure 8A surgical data network 201 is shown that includes a modular communication hub 203 that is configured to connect modular devices located in one or more operating rooms of a medical facility or any room in a medical facility that is specifically equipped for surgical operations to a cloud-based system (e.g., a cloud 204 that may include a remote server 213 coupled to a storage device 205). In one aspect, the modular communication hub 203 includes a network hub 207 and / or a network switch 209 that communicates with a network router. The modular communication hub 203 may also be coupled to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured to be passive, intelligent, or switched. A passive surgical data network acts as a conduit for data, enabling it to be transferred from one device (or segment) to another device (or segment) and cloud computing resources. An intelligent surgical data network includes additional features to enable monitoring of traffic passing through the surgical data network and configuration of each port in the network hub 207 or network switch 209. The intelligent surgical data network may be referred to as a manageable hub or switch. The switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0237] Modular devices 1a-1n located in an operating room may be coupled to a modular communication hub 203. A network hub 207 and / or a network switch 209 may be coupled to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transmitted to a cloud-based computer via a router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transmitted to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be coupled to a network switch 209. A network switch 209 may be coupled to a network hub 207 and / or a network router 211 to connect devices 2a-2m to a cloud 204. Data associated with devices 2a-2n may be transmitted to a cloud 204 via a network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transmitted to a local computer system 210 for local data processing and manipulation.

[0238] It should be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. The modular communication hub 203 can be included in a modular control tower that is configured to accommodate multiple devices 1a-1n / 2a-2m. A local computer system 210 can also be included in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images obtained by some of the devices 1a-1n / 2a-2m during a surgical procedure, for example. In various aspects, devices 1a-1n / 2a-2m may include, for example, various modules, such as an imaging module 138 coupled to an endoscope, a generator module 140 coupled to an energy-based surgical device, a smoke evacuation module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module in other modular devices that can be connected to a modular communication hub 203 of a surgical data network 201.

[0239] In one aspect, the surgical data network 201 may include a combination of (one or more) network hubs, (one or more) network switches, and (one or more) network routers that connect the devices 1a-1n / 2a-2m to the cloud. Any or all of the devices 1a-1n / 2a-2m coupled to the network hub or network switch can collect data in real time and transmit the data to the cloud computer for data processing and manipulation. It should be understood that cloud computing relies on shared computing resources rather than using local servers or personal devices to process software applications. The term "cloud" can be used as a metaphor for "Internet", although the term is not limited to this. Therefore, the term "cloud computing" can be used herein to refer to "a type of Internet-based computing", in which different services (such as servers, storage, and applications) are delivered to a modular communication hub 203 and / or a computer system 210 located in a surgical room (e.g., a fixed, mobile, temporary, or on-site operating room or space) and a device connected to the modular communication hub 203 and / or the computer system 210 via the Internet. The cloud infrastructure can be maintained by a cloud service provider. In this case, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services may perform a large number of calculations based on data collected by smart surgical instruments, robots, and other computerized devices located in the operating room. Hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0240] Applying cloud computer data processing technology to the data collected by the devices 1a-1n / 2a-2m, the surgical data network provides improved surgical results, reduced costs and improved patient satisfaction. At least some of the devices 1a-1n / 2a-2m can be used to observe tissue status to evaluate leakage or perfusion of sealed tissue after tissue sealing and cutting procedures. At least some of the devices 1a-1n / 2a-2m can be used to identify pathology, such as the effects of disease, using cloud-based computing to examine data including images of body tissue samples for diagnostic purposes. This includes positioning and edge confirmation of tissue and phenotype. At least some of the devices 1a-1n / 2a-2m can be used to identify the anatomical structure of the body using various sensors integrated with imaging devices and technologies (such as overlapping images captured by multiple imaging devices). The data (including image data) collected by the devices 1a-1n / 2a-2m can be transmitted to the cloud 204 or the local computer system 210 or both for data processing and manipulation, including image processing and manipulation. The data can be analyzed to improve surgical outcomes by determining whether further treatment can be pursued, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions. Such data analysis can be further processed using outcome analysis, and using standardized methods can provide helpful feedback to confirm surgical treatments and surgeon behavior, or suggest modifications to surgical treatments and surgeon behavior.

[0241] In one specific implementation, operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired channel or a wireless channel, depending on the configuration of the devices 1a-1n to the network hub. In one aspect, the network hub 207 may be implemented as a local network broadcast device that operates on the physical layer of the open system interconnection (OSI) model. The network hub provides connections to devices 1a-1n located in the same operating room network. The network hub 207 collects data in the form of packets and sends it to a router in half-duplex mode. The network hub 207 does not store any media access control / Internet protocol (MAC / IP) for transmitting device data. Only one of the devices 1a-1n can send data through the network hub 207 at a time. The network hub 207 has no knowledge of where to send information and broadcasts all network data on each connection and to the remote server 213 ( Fig. 9 ) Routing tables or intelligence that broadcasts all network data. The network hub 207 can detect basic network errors such as collisions, but broadcasting all information to multiple ports can pose security risks and cause bottlenecks.

[0242] In another specific implementation, the operating room devices 2a-2m can be connected to the network switch 209 through a wired channel or a wireless channel. The network switch 209 operates in the data link layer of the OSI model. The network switch 209 is a multicast device for connecting the devices 2a-2m located in the same operating room to the network. The network switch 209 sends data to the network router 211 in the form of a frame and operates in full-duplex mode. Multiple devices 2a-2m can send data simultaneously through the network switch 209. The network switch 209 stores and uses the MAC addresses of the devices 2a-2m to transmit data.

[0243] The network hub 207 and / or the network switch 209 are coupled to a network router 211 to connect to the cloud 204. The network router 211 operates in the network layer of the OSI model. The network router 211 creates a route for transmitting data packets received from the network hub 207 and / or the network switch 211 to cloud-based computer resources to further process and manipulate data collected by any or all of the devices 1a-1n / 2a-2m. The network router 211 can be used to connect two or more different networks located in different locations, such as, for example, different operating rooms in the same medical facility or different networks located in different operating rooms in different medical facilities. The network router 211 sends data to the cloud 204 in the form of packets and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transmit data.

[0244] In one example, the network hub 207 may be implemented as a USB hub that allows multiple USB devices to be connected to a host computer. The USB hub may expand a single USB port to multiple levels so that more ports are available to connect devices to a host system computer. The network hub 207 may include wired or wireless capabilities for receiving information through a wired channel or a wireless channel. In one aspect, a wireless USB short-range, high-bandwidth wireless radio communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in an operating room.

[0245] In other examples, the operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via the Bluetooth wireless technology standard for exchanging data from fixed and mobile devices over short distances (using short wavelength UHF radio waves in the ISM band of 2.4 to 2.485 GHz) and building a personal area network (PAN). In other aspects, the operating room devices 1a-1n / 2a-2m can communicate with the modular communication hub 203 via a variety of wireless or wired communication standards or protocols, including but not limited to Wi-Fi (IEEE 802.11 series), WiMAX (IEEE 802.16 series), IEEE802.20, Long Term Evolution (LTE) and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and Ethernet derivatives thereof, and any other wireless and wired protocols designated as 3G, 4G, 5G and above. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communications such as Wi-Fi and Bluetooth, and the second communication module may be dedicated to longer-range wireless communications such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, etc.

[0246] The modular communication hub 203 can serve as a central connection for one or all of the operating room devices 1a-1n / 2a-2m and handles data types called frames. The frames carry data generated by the devices 1a-1n / 2a-2m. When the modular communication hub 203 receives a frame, it is amplified and transmitted to the network router 211, which transmits the data to the cloud computing resources using multiple wireless or wired communication standards or protocols as described herein.

[0247] Modular communication hub 203 can be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Modular communication hub 203 is generally easy to install, construct and maintain, making it a good option for networking operating room devices 1a-1n / 2a-2m.

[0248] Fig. 9A computer-implemented interactive surgical system 200 is shown. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to the surgical system 102. Each surgical system 202 includes at least one surgical hub 206 in communication with a cloud 204, which may include a remote server 213. In one aspect, the computer-implemented interactive surgical system 200 includes a modular control tower 236 that is connected to a plurality of operating room devices, such as, for example, smart surgical instruments, robots, and other computerized devices located in the operating room. Fig.10 As shown in FIG, modular control tower 236 includes modular communication hub 203 coupled to computer system 210. Fig. 9 As shown in the example of , the modular control tower 236 is coupled to an imaging module 238 coupled to an endoscope 239, a generator module 240 coupled to an energy device 241, a smoke evacuator module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 optionally coupled to a display 237, and a non-contact sensor module 242. The operating room devices are coupled to cloud computing resources and data storage via the modular control tower 236. The robotic hub 222 can also be connected to the modular control tower 236 and cloud computing resources. The devices / instruments 235, the visualization system 208, etc. can be coupled to the modular control tower 236 via wired or wireless communication standards or protocols, as described herein. The modular control tower 236 can be coupled to a hub display 215 (e.g., monitor, screen) to display and overlay images received from the imaging module, device / instrument display, and / or other visualization system 208. The hub display may also combine graphics and overlay images to display data received from devices connected to the modular control tower.

[0249] Fig.10 A surgical hub 206 is shown comprising a plurality of modules coupled to a modular control tower 236. The modular control tower 236 comprises a modular communication hub 203 (e.g., network connectivity means) and a computer system 210 to provide, for example, local processing, visualization, and imaging. Fig.10 As shown in , the modular communication hub 203 can hierarchically construct connections to expand the number of modules (e.g., devices) that can be connected to the modular communication hub 203 and transmit data associated with the modules to the computer system 210, cloud computing resources, or both. Fig.10As shown in , each of the network hubs / switches in the modular communication hub 203 includes three downstream ports and one upstream port. The upstream network hub / switch is connected to the processor to provide communication connections with cloud computing resources and local display 217. Communication with the cloud 204 can be carried out through wired or wireless communication channels.

[0250] The surgical hub 206 employs a non-contact sensor module 242 to measure the size of the operating room and generates a map of the surgical room using an ultrasound or laser-based non-contact measurement device. The ultrasound-based non-contact sensor module scans the operating room by transmitting a burst of ultrasound waves and receiving the echoes as they bounce off the walls of the operating room, as described in U.S. Provisional Patent Application Serial No. 62 / 611,341, entitled “INTERACTIVE SURGICAL PLATFORM,” filed on December 28, 2017, under the title “Surgical Hub Spatial Awareness Within an Operating Room,” which is incorporated herein by reference in its entirety, wherein the sensor module is configured to be able to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module scans the operating room by transmitting laser pulses, receiving laser pulses that bounce off the walls of the operating room, and comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0251] The computer system 210 includes a processor 244 and a network interface 245. The processor 244 is coupled to a communication module 247, a storage device 248, a memory 249, a non-volatile memory 250, and an input / output interface 251 via a system bus. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus using any of a variety of available bus architectures including, but not limited to, a 9-bit bus, Industry Standard Architecture (ISA), Micro Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Small Computer System Interface (SCSI), or any other peripheral bus.

[0252] Processor 244 may be any single-core or multi-core processor, such as those offered by Texas Instruments under the trade name ARM Cortex. In one aspect, the processor may be a commercially available ARM Cortex-M4F processor core, such as the Texas Instruments LM4F230H5QR, which includes 256KB of single-cycle flash or other non-volatile memory (up to 40MHz) on-chip memory, a pre-fetch buffer for improved performance above 40MHz, 32KB of single-cycle sequential random access memory (SRAM), a 32KB MCU loaded with Internal read-only memory (ROM) for software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse width modulation (PWM) modules, one or more quadrature encoder input (QEI) simulations, one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels, details of which can be found in the product data sheet.

[0253] In one aspect, the processor 244 may include a safety controller including two families based on controllers such as TMS570 and RM4x, known under the trade name Hercules ARM Cortex R4 also produced by Texas Instruments. The safety controller may be configured to be capable of being specifically used for IEC 61508 and ISO26262 safety critical applications, among others, to provide advanced integrated safety features while delivering scalable performance, connectivity, and memory options.

[0254] System memory includes volatile memory and non-volatile memory. The basic input / output system (BIOS), which contains basic routines such as transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory may include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory includes random access memory (RAM) that acts as external cache memory. In addition, RAM is available in a variety of forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM) enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM) and direct Rambus RAM (DRRAM).

[0255] Computer system 210 also includes removable / non-removable, volatile / non-volatile computer storage media, such as, for example, disk storage. Disk storage includes, but is not limited to, devices such as disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage may include storage media individually or in combination with other storage media, including but not limited to optical disk drives such as optical disk ROM devices (CD-ROM), optical disk recordable drives (CD-R drives), optical disk rewritable drives (CD-RW drives), or digital universal disk ROM drives (DVD-ROM). In order to facilitate the connection of the disk storage device to the system bus, a removable or non-removable interface may be used.

[0256] It should be understood that the computer system 210 includes software that acts as an intermediary between a user and the basic computer resources described in a suitable operating environment. Such software includes an operating system. An operating system that can be stored on a disk storage device is used to control and allocate the resources of the computer system. System applications utilize the operating system to manage resources through program modules and program data stored in system memory or disk storage devices. It should be understood that the various components described herein can be implemented with various operating systems or combinations of operating systems.

[0257] The user inputs commands or information into the computer system 210 through (one or more) input devices coupled to the I / O interface 251. Input devices include, but are not limited to, pointing devices such as a mouse, a touch ball, a stylus, a touch pad, a keyboard, a microphone, a joystick, a game pad, a satellite dish, a scanner, a TV tuner card, a digital camera, a digital video camera, a web camera, etc. These and other input devices are connected to the processor through the system bus via (one or more) interface ports. (One or more) interface ports include, for example, serial ports, parallel ports, game ports, and USB. (One or more) output devices use the same type of port as (one or more) input devices. Thus, for example, a USB port can be used to provide input to the computer system and output information from the computer system to an output device. Output adapters are provided to illustrate that there are some output devices (such as monitors, displays, speakers, and printers) that require special adapters among other output devices. Output adapters include, by way of example, but are not limited to, video and sound cards that provide connection means between output devices and the system bus. It should be noted that other devices or systems of devices such as (one or more) remote computers provide both input capabilities and output capabilities.

[0258] The computer system 210 can operate in a networked environment using a logical connection with one or more remote computers (such as (one or more) cloud computers) or local computers. (One or more) remote cloud computers can be personal computers, servers, routers, network PCs, workstations, microprocessor-based appliances, peer devices or other public network nodes, etc., and generally include many or all of the elements described relative to the computer system. For simplicity, only a memory storage device with (one or more) remote computers is shown. (One or more) remote computers are logically connected to the computer system through a network interface, and then physically connected via a communication connection. The network interface covers communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include fiber distributed data interface (FDDI), copper distributed data interface (CDDI), Ethernet / IEEE 802.3, token ring / IEEE 802.5, etc. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks such as integrated services digital networks (ISDN) and their variants, packet switching networks, and digital subscriber lines (DSL).

[0259] In various aspects, Fig.10 computer system 210, imaging module 238 and / or visualization system 208, and / or Figure 9-10 The processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) for processing digital images. The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instructions, multiple data (MIMD) techniques to improve speed and efficiency. The digital image processing engine may perform a range of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0260] The communication connection(s) refers to the hardware / software used to connect the network interface to the bus. Although the communication connections are shown for exemplary clarification within the computer system, they may also be located external to the computer system 210. The hardware / software necessary to connect to the network interface includes internal and external technologies such as modems, including conventional telephone grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards for exemplary purposes only.

[0261] In all respects, reference Figures 9 and 10 The described device / instrument 235 may be implemented as a surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ). Therefore, surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) is configured to interface with the module control tower 236 and the surgical hub 206. Once connected to the surgical hub 206, the surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) is configured to be able to interface with the cloud 204, server 213, other hub-connected instruments, hub display 215, or visualization system 209, or a combination thereof. In addition, once connected to the hub 206, the surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) can utilize the processing circuitry available in the hub local computer system 210.

[0262] Fig.11 A functional block diagram of one aspect of a USB network hub 300 device according to at least one aspect of the present disclosure is shown. In the illustrated aspect, the USB network hub device 300 employs the TUSB2036 integrated circuit hub of Texas Instruments. The USB network hub 300 is a CMOS device that provides an upstream USB transceiver port 302 and up to three downstream USB transceiver ports 304, 306, 308 according to the USB 2.0 specification. The upstream USB transceiver port 302 is a differential root data port that includes a differential data negative (DP0) input paired with a differential data positive (DM0) input. The three downstream USB transceiver ports 304, 306, 308 are differential data ports, each of which includes a differential data positive (DP1-DP3) output paired with a differential data negative (DM1-DM3) output.

[0263] The USB network hub 300 device is implemented with a digital state machine rather than a microcontroller, and does not require firmware programming. A fully compatible USB transceiver is integrated into the circuit for the upstream USB transceiver port 302 and all downstream USB transceiver ports 304, 306, 308. The downstream USB transceiver ports 304, 306, 308 support both full-speed devices and low-speed devices by automatically setting the conversion rate according to the speed of the device attached to the port. The USB network hub 300 device can be configured to be in bus-powered mode or self-powered mode, and includes a hub power logic 312 for managing power.

[0264] The USB network hub 300 device includes a serial interface engine 310 (SIE). The SIE 310 is the front end of the USB network hub 300 hardware and handles most of the protocols described in Chapter 8 of the USB specification. The SIE 310 generally includes up to transaction-level signaling. The functions it handles may include: packet identification, transaction ordering, SOP, EOP, RESET and RESUME signal detection / generation, clock / data separation, non-return to zero inversion (NRZI) data encoding / decoding and digit filling, CRC generation and verification (token and data), packet ID (PID) generation and verification / decoding, and / or serial-to-parallel / parallel-to-serial conversion. The 310 receives a clock input 314 and is coupled to a pause / resume logic and frame timer 316 circuit and a hub repeater circuit 318 to control the communication between the upstream USB transceiver port 302 and the downstream USB transceiver port 304, 306, 308 through the port logic circuit 320, 322, 324. SIE 310 is coupled to command decoder 326 via interface logic 328 to control commands from the serial EEPROM via serial EEPROM interface 330 .

[0265] In various aspects, the USB network hub 300 can connect 127 functions constructed in up to six logical layers (hierarchies) to a single computer. In addition, the USB network hub 300 can be connected to all external devices using a standardized four-wire cable that provides both communication and power distribution. The power configuration is a bus-powered mode and a self-powered mode. The USB network hub 300 can be configured to support four power management modes: a bus-powered hub with individual port power management or a complete set of port power management, and a self-powered hub with individual port power management or a complete set of port power management. In one aspect, the USB network hub 300 and the upstream USB transceiver port 302 are inserted into the USB host controller using a USB cable, and the downstream USB transceiver ports 304, 306, 308 are exposed for connecting USB compatible devices, etc.

[0266] Additional details regarding the structure and functionality of a surgical hub and / or a network of surgical hubs may be found in U.S. Provisional Patent Application No. 62 / 659,900, filed on April 19, 2018, entitled “METHOD OF HUB COMMUNICATION,” which is incorporated herein by reference in its entirety.

[0267] Cloud system hardware and functional modules

[0268] Fig.12 1 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system is configured to monitor and analyze data related to the operation of various surgical systems, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system includes a cloud-based analysis system. Although the cloud-based analysis system is described as a surgical system, it is not necessarily so limited and can generally be a cloud-based medical system. Fig.12 As shown, the cloud-based analysis system includes a plurality of surgical instruments 7012 (which may be the same or similar to instrument 112), a plurality of surgical hubs 7006 (which may be the same or similar to hub 106), and a surgical data network 7001 (which may be the same or similar to network 201) to couple the surgical hubs 7006 to a cloud 7004 (which may be the same or similar to cloud 204). Each of the plurality of surgical hubs 7006 is communicatively coupled to one or more surgical instruments 7012. The hub 7006 is also communicatively coupled to a cloud 7004 of a computer-implemented interactive surgical system via the network 7001. The cloud 7004 is a remote centralized hardware and software source for storing, manipulating, and transmitting data generated based on the operation of various surgical systems. As shown in FIG. Fig.12 As shown, access to cloud 7004 is achieved via network 7001, which can be the Internet or some other suitable computer network. A surgical hub 7006 coupled to cloud 7004 can be considered the client side of a cloud computing system (i.e., a cloud-based analysis system). Surgical instruments 7012 are paired with surgical hub 7006 for controlling and implementing various surgical procedures or operations as described herein.

[0269] In addition, the surgical instrument 7012 may include a transceiver for transmitting data to and from its corresponding surgical hub 7006 (which may also include a transceiver). The combination of the surgical instrument 7012 and the corresponding hub 7006 may indicate a specific location for providing a medical procedure, such as an operating room in a medical facility (e.g., a hospital). For example, the memory of the surgical hub 7006 may store location data. Fig.12 As shown, cloud 7004 includes a central server 7013 (which can be connected to Figure 1 The remote server 113 and / or Fig. 9 7006 ), a hub application server 7002, a data analysis module 7034, and an input / output ("I / O") interface 7007. The central servers 7013 of the cloud 7004 collectively host a cloud computing system that includes monitoring requests from client surgical hubs 7006 and managing the processing capacity of the cloud 7004 for executing the requests. Each of the central servers 7013 includes one or more processors 7008 coupled to a suitable memory device 7010, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. The memory device 7010 may include machine executable instructions that, when executed, cause the processor 7008 to execute the data analysis module 7034 for cloud-based data analysis, operations, recommendations, and other operations described below. In addition, the processor 7008 may execute the data analysis module 7034 independently or in conjunction with a hub application independently executed by the hub 7006. The central server 7013 also includes a database 2212 of aggregated medical data that may reside in the memory 2210 .

[0270] Based on the connection to various surgical hubs 7006 via the network 7001, the cloud 7004 can aggregate data from specific data generated by various surgical instruments 7012 and their corresponding hubs 7006. Such aggregated data can be stored in the aggregated medical database 7011 of the cloud 7004. In particular, the cloud 7004 can advantageously perform data analysis and operations on the aggregated data to generate insights and / or perform functions that cannot be achieved by the individual hubs 7006 themselves. To this end, as Fig.12As shown, cloud 7004 and surgical hub 7006 are communicatively coupled to transmit and receive information. I / O interface 7007 is connected to multiple surgical hubs 7006 via network 7001. In this way, I / O interface 7007 can be configured to transmit information between surgical hub 7006 and database 7011 of aggregated medical data. Therefore, I / O interface 7007 can facilitate read / write operations of cloud-based analysis systems. Such read / write operations can be performed in response to requests from hub 7006. These requests can be transmitted to hub 7006 by hub application. I / O interface 7007 may include one or more high-speed data ports, which may include universal serial bus (USB) ports, IEEE 1394 ports, and Wi-Fi and Bluetooth I / O interfaces for connecting cloud 7004 to hub 7006. The hub application server 7002 of the cloud 7004 is configured to host and provide shared capabilities to software applications (e.g., hub applications) executed by the surgical hub 7006. For example, the hub application server 7002 can manage requests made by the hub application through the hub 7006, control access to the database 7011 of aggregated medical data, and perform load balancing. Fig.13 The data analysis module 7034 is described in more detail.

[0271] The specific cloud computing system configurations described in the present disclosure are specifically designed to address various issues that arise in the context of medical procedures and surgeries performed using medical devices, such as surgical instruments 7012, 112. Specifically, the surgical instrument 7012 may be a digital surgical device that is configured to be able to interact with the cloud 7004 for implementing technologies that improve the performance of surgical procedures. The various surgical instruments 7012 and / or the surgical hub 7006 may include a touch-controlled user interface so that the clinician can control aspects of the interaction between the surgical instrument 7012 and the cloud 7004. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0272] Fig.13 7004 is a block diagram illustrating the functional architecture of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. The cloud-based analysis system includes a plurality of data analysis modules 7034 that can be executed by a processor 7008 of the cloud 7004 to provide data analysis solutions for problems that arise specifically in the medical field. Fig.13As shown, the functionality of the cloud-based data analysis module 7034 may be facilitated via a hub application 7014 hosted by a hub application server 7002, which may be accessed on a surgical hub 7006. The cloud processor 7008 and the hub application 7014 may operate in conjunction to execute the data analysis module 7034. An application program interface (API) 7016 defines a set of protocols and routines corresponding to the hub application 7014. In addition, the API 7016 manages the storage and retrieval of data to and from a database 7011 of aggregated medical data for operation of the application 7014. A cache 7018 also stores data (e.g., temporarily) and is coupled to the API 7016 for more efficient retrieval of data used by the application 7014. Fig.13 The data analysis module 7034 in the cloud 7004 includes a resource optimization module 7020, a data collection and aggregation module 7022, an authentication and security module 7024, a control program update module 7026, a patient outcome analysis module 7028, a recommendation module 7030, and a data classification and prioritization module 7032. According to some aspects, the cloud 7004 can also implement other suitable data analysis modules. In one aspect, the data analysis module is used for specific recommendations based on analysis trends, results, and other data.

[0273] For example, the data collection and aggregation module 7022 can be used to generate self-describing data (e.g., metadata), including identification of significant features or configurations (e.g., trends), management of redundant data sets, and storage of data in paired data sets, which can be grouped by surgical operation, but not necessarily locked to the actual surgical date and surgeon. Specifically, the paired data sets generated by the operation of the surgical instrument 7012 may include applying binary classification, for example, bleeding or non-bleeding events. More generally, the binary classification may be characterized as a desired event (e.g., a successful surgical operation) or an undesirable event (e.g., a misfired or misused surgical instrument 7012). Aggregated self-describing data may correspond to individual data received from various groups or subgroups of the surgical hub 7006. Therefore, the data collection and aggregation module 7022 may generate aggregated metadata or other organizational data based on the raw data received from the surgical hub 7006. To this end, the processor 7008 may be operably coupled to the hub application 7014 and the database 7011 of aggregated medical data for executing the data analysis module 7034. The data collection and aggregation module 7022 may store the aggregated organizational data into the database 2212 of aggregated medical data.

[0274] The resource optimization module 7020 may be configured to analyze the aggregated data to determine the optimal use of resources for a particular or a group of medical facilities. For example, the resource optimization module 7020 may determine the optimal sequence point of the surgical stapling instrument 7012 for a group of medical facilities based on the corresponding predicted demand of such instruments 7012. The resource optimization module 7020 may also evaluate the resource usage or other operational configurations of various medical facilities to determine whether resource usage can be improved. Similarly, the recommendation module 7030 may be configured to analyze the aggregated organizational data from the data collection and aggregation module 7022 to provide recommendations. For example, the recommendation module 7030 may recommend to a medical facility (e.g., a medical service provider, such as a hospital) that a specific surgical instrument 7012 should be upgraded to an improved version based on, for example, a higher than expected error rate. In addition, the recommendation module 7030 and / or the resource optimization module 7020 may recommend better supply chain parameters, such as product reordering points, and provide suggestions for different surgical instruments 7012, their use or surgical steps to improve surgical results. Medical facilities may receive such recommendations via the corresponding surgical hub 7006. More specific recommendations may also be provided regarding parameters or configurations of various surgical instruments 7012. The hub 7006 and / or the surgical instruments 7012 may also each have a display screen that displays data or recommendations provided by the cloud 7004.

[0275] The patient outcome analysis module 7028 can analyze surgical results associated with the currently used operating parameters of the surgical instrument 7012. The patient outcome analysis module 7028 can also analyze and evaluate other potential operating parameters. In this regard, the recommendation module 7030 can recommend the use of these other potential operating parameters based on producing better surgical results (such as better sealing or less bleeding). For example, the recommendation module 7030 can transmit to the surgical hub 7006 a recommendation on when to use a specific bin for the corresponding suture surgical instrument 7012. Therefore, the cloud-based analysis system can be configured to be able to analyze a large amount of raw data collection and provide centralized recommendations for multiple medical facilities (advantageously determined based on aggregated data) when controlling common variables. For example, the cloud-based analysis system can analyze, evaluate and / or aggregate data based on the type of medical practice, the type of patient, the number of patients, the geographical similarity between medical providers, which medical providers / facilities use similar types of instruments, etc., so that any single medical facility alone cannot be analyzed independently.

[0276] The control program update module 7026 may be configured to be able to perform various surgical instrument 7012 recommendations when the corresponding control program is updated. For example, the patient outcome analysis module 7028 may identify correlations that link specific control parameters to successful (or unsuccessful) results. Such correlations may be resolved when the updated control program is transmitted to the surgical instrument 7012 via the control program update module 7026. Updates to the instrument 7012 transmitted via the corresponding hub 7006 may be combined with aggregated performance data collected and analyzed by the data collection and aggregation module 7022 of the cloud 7004. In addition, the patient outcome analysis module 7028 and the recommendation module 7030 may identify improved methods of using the instrument 7012 based on the aggregated performance data.

[0277] The cloud-based analysis system may include security features implemented by the cloud 7004. These security features may be managed by the authorization and security module 7024. Each surgical hub 7006 may have associated unique credentials, such as a username, password, and other suitable security credentials. These credentials may be stored in the memory 7010 and associated with the cloud access level allowed. For example, based on providing accurate credentials, the surgical hub 7006 may be granted access rights to communicate with the cloud to a predetermined extent (e.g., may only participate in transmitting or receiving certain limited types of information). To this end, the database 7011 of the aggregated medical data of the cloud 7004 may include a database of authorized credentials for verifying the accuracy of the credentials provided. Different credentials may be associated with different levels of authority to interact with the cloud 7004, such as a predetermined access level for receiving data analysis generated by the cloud 7004.

[0278] In addition, for security purposes, the cloud may maintain a database of hubs 7006, instruments 7012, and other devices that may include a "blacklist" of prohibited devices. Specifically, surgical hubs 7006 listed on the blacklist may not be allowed to interact with the cloud, while surgical instruments 7012 listed on the blacklist may not have functional access to the corresponding hub 7006 and / or may be blocked from fully functioning when paired with their corresponding hub 7006. In addition or alternatively, the cloud 7004 may flag instruments 7012 based on incompatibility or other specified criteria. In this way, counterfeit medical devices and improper reuse of such devices throughout the cloud-based analysis system can be identified and addressed.

[0279] The surgical instrument 7012 may use a wireless transceiver to transmit a wireless signal, which may represent, for example, an authorization credential for accessing the corresponding hub 7006 and the cloud 7004. A wired transceiver may also be used to transmit the signal. Such authorization credentials may be stored in a corresponding memory device of the surgical instrument 7012. The authorization and security module 7024 may determine whether the authorization credential is accurate or forged. The authorization and security module 7024 may also dynamically generate authorization credentials for enhanced security. The credential may also be encrypted, such as by using hash-based encryption. Upon transmitting the appropriate authorization, the surgical instrument 7012 may transmit a signal to the corresponding hub 7006 and ultimately to the cloud 7004 to indicate that the instrument 7012 is ready to acquire and transmit medical data. In response, the cloud 7004 may be transformed into a state capable of receiving medical data for storage in a database 7011 of aggregated medical data. The data transmission readiness may be indicated, for example, by a light indicator on the instrument 7012. The cloud 7004 may also transmit a signal to the surgical instrument 7012 for updating its associated control program. The cloud 7004 can transmit signals related to a particular class of surgical instruments 7012 (e.g., electrosurgical instruments) so that software updates to the control program are transmitted only to the appropriate surgical instruments 7012. In addition, the cloud 7004 can be used to implement system-wide solutions to solve local or global problems based on selective data transmission and authorization credentials. For example, if a group of surgical instruments 7012 are identified as having a common manufacturing defect, the cloud 7004 can change the authorization credentials corresponding to the group to implement an operational lockout for the group.

[0280] The cloud-based analysis system can allow monitoring of multiple medical facilities (e.g., medical facilities such as hospitals) to determine improved practices and accordingly suggest changes (e.g., via the suggestion module 2030). Therefore, the processor 7008 of the cloud 7004 can analyze the data associated with each medical facility to identify the facility and aggregate the data with other data associated with other medical facilities in the group. For example, a group can be defined based on similar operating practices or geographic locations. In this way, the cloud 7004 can provide analysis and suggestions within the scope of the medical facility group. The cloud-based analysis system can also be used to enhance situational awareness. For example, the processor 7008 can predictively model the impact of the recommendation on the cost and effectiveness of a particular facility (relative to overall operations and / or various medical procedures). The cost and effectiveness associated with the particular facility can also be compared with the corresponding local areas of other facilities or any other comparable facilities.

[0281] The data classification and prioritization module 7032 can prioritize and classify the data based on criticality (e.g., severity, unexpectedness, suspicion of the medical event associated with the data). This sorting and prioritization can be used in conjunction with the functions of the other data analysis modules 7034 described above to improve the cloud-based analysis and operations described herein. For example, the data sorting and prioritization module 7032 can assign priorities to the data analysis performed by the data collection and aggregation module 7022 and the patient outcome analysis module 7028. Different priority levels can cause specific responses from the cloud 7004 (corresponding to the level of urgency), such as escalation of rapid response, special treatment, exclusion from the database 7011 of aggregated medical data, or other appropriate responses. In addition, if necessary, the cloud 7004 can transmit a request for additional data from the corresponding surgical instrument 7012 through the hub application server (e.g., a push message). The push message can cause a notification displayed on the corresponding hub 7006 for requesting support or additional data. The push message may be required in the case where the cloud detects significant irregularity or abnormality and the cloud cannot determine the cause of the irregularity. The central server 7013 may be programmed to trigger this push message in certain significant circumstances, such as, for example, when the data is determined to be different from an expected value exceeding a predetermined threshold or when it appears that safety has been included.

[0282] In all respects, the above reference Fig.12 and Fig.13 The surgical instrument 7012 described may be implemented as a surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ). Therefore, surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) is configured to interface with the surgical hub 7006 and the network 2001, which is configured to interface with the cloud 7004. Thus, the processing power provided by the central server 7013 and the data analysis module 7034 is configured to process data from the surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) information (e.g., data and control).

[0283] Additional details about the cloud analytics system may be found in U.S. Provisional Patent Application No. 62 / 659,900, filed on April 19, 2018, entitled “METHOD OF HUB COMMUNICATION,” which is incorporated herein by reference in its entirety.

[0284] Situational Awareness

[0285] Although a "smart" device including a control algorithm responsive to sensed data may be an improvement over a "dumb" device that operates without regard to the sensed data, some sensed data may be incomplete or uncertain when considered in isolation, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the surgical context (e.g., knowing the type of tissue being operated on or the type of surgery being performed), the control algorithm may incorrectly or suboptimally control the modular device given a specific background-free sensed data. For example, the best way to control a control algorithm for a surgical instrument in response to a specific sensed parameter may vary depending on the specific tissue type being operated on. This is due to the fact that different tissue types have different properties (e.g., tear resistance) and therefore respond differently to actions taken by a surgical instrument. Therefore, it may be desirable for a surgical instrument to take different actions even when sensing the same measurement for a specific parameter. As a specific example, the best way to control a surgical suturing and cutting instrument in response to the instrument sensing an unexpectedly high force used to close its end effector will vary depending on whether the tissue type is easy to tear or tear-resistant. For tissue that is prone to tearing (such as lung tissue), the control algorithm of the instrument will optimally gradually reduce the motor speed in response to the unexpectedly high force for closing, thereby avoiding tearing the tissue. For tissue that is resistant to tearing (such as stomach tissue), the control algorithm of the instrument will optimally gradually increase the motor speed in response to the unexpectedly high force for closing, thereby ensuring that the end effector is properly clamped on the tissue. Without knowing whether lung tissue or stomach tissue has been clamped, the control algorithm may make suboptimal decisions.

[0286] One solution utilizes a surgical hub that includes a system that is configured to be able to derive information about a surgical procedure being performed based on data received from various data sources, and then control paired modular devices accordingly. In other words, the surgical hub is configured to be able to infer information about the surgical procedure from the received data, and then control modular devices paired with the surgical hub based on the context of the inferred surgical procedure. Fig.14A diagram of a situational awareness surgical system 5100 is shown in accordance with at least one aspect of the present disclosure. In some examples, data sources 5126 include, for example, modular apparatus 5102 (which may include sensors configured to detect parameters associated with a patient and / or the modular apparatus itself), database 5122 (e.g., an EMR database containing patient records), and patient monitoring devices 5124 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0287] The surgical hub 5104, which may be similar to the hub 106 in many respects, may be configured to be able to derive contextual information related to the surgical procedure from the data, for example, based on a particular combination of the received data or a particular order in which the data is received from the data source 5126. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity being the subject of the surgery. Some aspects of the surgical hub 5104 This ability to derive or infer information related to the surgical procedure from the received data may be referred to as "situational awareness." In one example, the surgical hub 5104 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 5104 that derives contextual information related to the surgical procedure from the received data.

[0288] The situational awareness system of the surgical hub 5104 can be configured to be able to derive context information from data received from the data source 5126 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or a machine learning system (e.g., an artificial neural network) that has been trained on training data to associate various inputs (e.g., data from the database 5122, the patient monitoring device 5124, and / or the modular device 5102) with corresponding context information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive context information about the surgical procedure from the provided input. In another example, the situational awareness system may include a lookup table that stores pre-characterized context information about the surgical procedure associated with one or more inputs (or input ranges) corresponding to the context information. In response to a query using one or more inputs, the lookup table may return the corresponding context information used by the situational awareness system to control the modular device 5102. In one example, the context information received by the situational awareness system of the surgical hub 5104 is associated with a specific control adjustment or a set of control adjustments for one or more modular devices 5102. In another example, the situational awareness system includes an additional machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 5102 when context information is provided as input.

[0289] The surgical hub 5104 incorporating a situational awareness system provides numerous benefits to the surgical system 5100. One benefit includes improved interpretation of sensed and collected data, which in turn improves processing accuracy and / or use of the data during the surgical procedure. Returning to the previous example, the situational awareness surgical hub 5104 can determine the type of tissue being operated on; thus, when an unexpectedly high force is detected for the end effector of a closing surgical instrument, the situational awareness surgical hub 5104 can appropriately ramp up or ramp down the motor speed of the surgical instrument for the tissue type.

[0290] As another example, the type of tissue being operated on can affect the adjustment of the compression rate and load threshold of the surgical stapling and cutting instrument for a specific tissue gap measurement. The situational awareness surgical hub 5104 can infer whether the surgical procedure being performed is a thoracic surgery or an abdominal surgery, thereby allowing the surgical hub 5104 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 5104 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument for the type of tissue.

[0291] As yet another example, the type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situational awareness surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the type of surgery. Since one type of surgery is typically performed in a particular body cavity, the surgical hub 5104 can then control the motor rate of the smoke evacuator appropriately for the body cavity in which the procedure is being performed. Thus, the situational awareness surgical hub 5104 can provide consistent smoke evacuation for both thoracic and abdominal surgeries.

[0292] As yet another example, the type of surgery being performed may affect the optimal energy level at which an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument operates. For example, arthroscopic surgery requires higher energy levels because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in a fluid. The situational awareness surgical hub 5104 may determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 5104 may then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on may affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The situational awareness surgical hub 5104 may determine the type of surgery being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the expected tissue profile of the surgery. In addition, the situational awareness surgical hub 5104 may be configured to be able to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the entire surgical procedure rather than just on a surgery-by-surgery basis. The situational awareness surgical hub 5104 can determine the step of a surgical procedure being performed or to be subsequently performed and then update the control algorithm for the generator and / or ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type based on the surgical step.

[0293] As yet another example, data may be extracted from additional data sources 5126 to improve conclusions extracted by the surgical hub 5104 from one data source 5126. The situational awareness surgical hub 5104 may augment the data it receives from the modular devices 5102 with contextual information about the surgical procedure that has been constructed from other data sources 5126. For example, the situational awareness surgical hub 5104 may be configured to be able to determine whether hemostasis has occurred (i.e., whether bleeding at a surgical site has stopped) based on video or image data received from a medical imaging device. However, in some cases, the video or image data may be inconclusive. Thus, in one example, the surgical hub 5104 may also be configured to be able to combine physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 5104) with visual or image data of hemostasis (e.g., from a medical imaging device 124 communicatively coupled to the surgical hub 5104) Figure 2 )) to determine the integrity of the suture or tissue weld. In other words, the situational awareness system of the surgical hub 5104 can consider physiological measurement data to provide additional context when analyzing visualization data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0294] Another benefit includes actively and automatically controlling the paired modular devices 5102 according to the specific steps of the surgical procedure being performed to reduce the number of times medical personnel need to interact with or control the surgical system 5100 during the surgical procedure. For example, if the situational awareness surgical hub 5104 determines that a subsequent step of the procedure requires the use of an RF electrosurgical instrument, it can actively activate the generator connected to the instrument. Actively activating the energy source allows the instrument to be ready for use as soon as the previous step of the procedure is completed.

[0295] As another example, the situational awareness surgical hub 5104 may determine whether the current or subsequent steps of the surgical procedure require a different view or degree of magnification on the display based on the feature(s) that the surgeon anticipates needing to view at the surgical site. The surgical hub 5104 may then proactively change the displayed view (e.g., provided by a medical imaging device for the visualization system 108) accordingly, such that the display is automatically adjusted throughout the surgical procedure.

[0296] As yet another example, the situational awareness surgical hub 5104 can determine which step of the surgical procedure is being performed or will be performed subsequently and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 5104 can be configured to automatically call up data screens based on the step of the surgical procedure being performed without having to wait for the surgeon to request that specific information.

[0297] Another benefit includes checking for errors during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 5104 can determine whether the operating room is correctly or optimally set up for the surgical procedure to be performed. The surgical hub 5104 can be configured to be able to determine the type of surgical procedure being performed, retrieve the corresponding list, product location, or setup requirements (e.g., from a memory), and then compare the current operating room layout with the standard layout for the type of surgical procedure being performed determined by the surgical hub 5104. In one example, the surgical hub 5104 can be configured to be able to compare a list of items for surgery (e.g., scanned by a suitable scanner) and / or a list of devices paired with the surgical hub 5104 with a recommended or expected list of items and / or devices for a given surgical procedure. The surgical hub 5104 can be configured to be able to provide an alert indicating that a specific modular device 5102, a patient monitoring device 5124, and / or other surgical items are missing if there is any discontinuity between the lists. In one example, the surgical hub 5104 can be configured to determine the relative distance or position of the modular device 5102 and the patient monitoring device 5124, for example, via a proximity sensor. The surgical hub 5104 can compare the relative position of the devices to a recommended or expected layout for a particular surgical procedure. The surgical hub 5104 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout if there is any discontinuity between the layouts.

[0298] As another example, the situational awareness surgical hub 5104 may determine whether a surgeon (or other medical personnel) is making an error or otherwise deviating from an expected course of action during a surgical procedure. For example, the surgical hub 5104 may be configured to be able to determine the type of surgical procedure being performed, retrieve a corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the surgical procedure to the expected steps or equipment determined by the surgical hub 5104 for the type of surgical procedure being performed. In one example, the surgical hub 5104 may be configured to be able to provide an alert indicating that an unexpected action is being performed or an unexpected device is being used at a particular step in the surgical procedure.

[0299] In general, the situational awareness system for the surgical hub 5104 improves surgical outcomes by adjusting surgical instruments (and other modular devices 5102) to the specific context of each surgical procedure (such as adjusting for different tissue types) and validating actions during the surgical procedure. The situational awareness system also improves the efficiency with which the surgeon performs the surgical procedure by automatically suggesting the next step, providing data, and adjusting displays and other modular devices 5102 in the operating room according to the specific context of the surgery.

[0300] In one aspect, the following reference Figures 16 to 22B As described above, the modular device 5102 can be implemented as a surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ). Therefore, it is implemented as surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) is configured to be able to serve as a data source 5126 and interact with a database 5122 and a patient monitoring device 5124. Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B )'s modular device 5102 is further configured to be capable of interacting with the surgical hub 5104 to provide information (e.g., data and control) to the surgical hub 5104 and to receive information (e.g., data and control) from the surgical hub 5104.

[0301] See now Fig.15 , which shows a depiction of a hub such as surgical hub 106 or 206 ( Figures 1 to 11 ). The timeline 5200 is an illustrative surgical procedure and background information that the surgical hub 106, 206 can derive from data received from data sources at each step in the surgical procedure. The timeline 5200 depicts typical steps that nurses, surgeons, and other medical personnel will take during a segmentectomy surgery, starting with setting up the operating room and ending with transferring the patient to a postoperative recovery room.

[0302] The situational awareness surgical hub 106, 206 receives data from data sources throughout the surgical procedure, including data generated each time a medical professional utilizes a modular device paired with the surgical hub 106, 206. The surgical hub 106, 206 can receive this data from paired modular devices and other data sources, and continually derive inferences (i.e., contextual information) about the ongoing surgery as new data is received, such as which step of the surgery is being performed at any given time. The situational awareness system of the surgical hub 106, 206 can, for example, record data related to the procedure for generating a report, verify the steps being taken by the medical professional, provide data or prompts that may be related to a particular procedure step (e.g., via a display screen), adjust the modular device based on the context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and take any other such actions described above.

[0303] As a first step 5202 in this exemplary operation, hospital staff retrieves the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hub 106, 206 determines that the operation to be performed is a thoracic operation.

[0304] In a second step 5204, the staff scans the incoming medical supplies for the procedure. The surgical hub 106, 206 cross-references the scanned supplies with a list of supplies used in various types of surgeries and confirms that the mix of supplies corresponds to a thoracic procedure. Additionally, the surgical hub 106, 206 can also determine that the procedure is not a wedge procedure (because the incoming supplies lack certain supplies required for a thoracic wedge procedure or otherwise do not correspond to a thoracic wedge procedure).

[0305] In a third step 5206, the medical personnel scans the patient belt via a scanner communicatively connected to the surgical hub 106, 206. The surgical hub 106, 206 may then confirm the identity of the patient based on the scanned data.

[0306] In the fourth step 5208, the medical staff turns on the auxiliary equipment. The auxiliary equipment utilized may vary depending on the type of surgical procedure and the technique to be used by the surgeon, but in this exemplary case, they include a smoke evacuator, an insufflator, and a medical imaging device. When activated, as part of its initialization process, the auxiliary equipment, which is a modular device, can be automatically paired with a surgical hub 106, 206 located in a specific vicinity of the modular device. The surgical hub 106, 206 can then derive background information about the surgical procedure by detecting the type of modular device paired with it during this preoperative phase or initialization phase. In this specific example, the surgical hub 106, 206 determines that the surgical procedure is a VATS procedure based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, a list of medical supplies used in the surgery, and the type of modular device connected to the hub, the surgical hub 106, 206 can generally infer the specific surgery that the surgical team will perform. Once the surgical hub 106, 206 knows what specific procedure is being performed, the surgical hub 106, 206 can retrieve the steps of that procedure from memory or the cloud and then cross-reference the data it subsequently receives from connected data sources (e.g., modular devices and patient monitoring devices) to infer what steps of the surgical procedure the surgical team is performing.

[0307] In a fifth step 5210, the staff attaches EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hub 106, 206. When the surgical hub 106, 206 begins to receive data from the patient monitoring devices, the surgical hub 106, 206 thus confirms that the patient is in the operating room.

[0308] In step 5212, the medical staff induces anesthesia in the patient. The surgical hub 106, 206 may infer that the patient is under anesthesia based on data from the modular device and / or the patient monitoring device (including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof). Upon completion of step 5212, the preoperative portion of the lung segmentectomy surgery is completed and the surgical portion begins.

[0309] In a seventh step 5214, collapse the patient's lung being operated on (while ventilation is switched to the contralateral lung). For example, the surgical hub 106, 206 may infer from the ventilator data that the patient's lung has collapsed. The surgical hub 106, 206 may infer that the surgical portion of the procedure has begun because it may compare the detection of the patient's lung collapsing to the expected steps of the procedure (which may have been previously accessed or retrieved) and determine that collapsing the lung is the first surgical step in this particular procedure.

[0310] In an eighth step 5216, a medical imaging device (e.g., an endoscope) is inserted and video from the medical imaging device is initiated. The surgical hub 106, 206 receives medical imaging device data (i.e., video or image data) through its connection to the medical imaging device. After receiving the medical imaging device data, the surgical hub 106, 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 106, 206 can determine that the particular procedure being performed is a segmentectomy, rather than a lobectomy (note that a wedge procedure has been ruled out based on the data received by the surgical hub 106, 206 at the second step 5204 of the procedure). From the medical imaging device 124 ( Figure 2 ) can be used to determine contextual information related to the type of procedure being performed in a variety of different ways, including by determining the angle of the medical imaging device relative to the visualization orientation of the patient's anatomy, monitoring the number of medical imaging devices utilized (i.e., activated and paired with the surgical hub 106, 206), and monitoring the type of visualization device utilized. For example, one technique for performing a VATS lobectomy places a camera in the lower anterior corner of the patient's thorax above the diaphragm, while a technique for performing a VATS segmentectomy places a camera in an anterior intercostal position relative to the segmental fissure. For example, using pattern recognition or machine learning techniques, a situational awareness system can be trained to recognize the positioning of the medical imaging device based on the visualization of the patient's anatomy. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As another example, a technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively coupled to the surgical hub as part of the visualization system) to visualize a segmental fissure that is not used in a VATS pulmonary resection. By tracking any or all of this data from the medical imaging device, the surgical hub 106, 206 can therefore determine the specific type of surgery being performed and / or the techniques used for a particular type of surgery.

[0311] In step 5218, the surgical team begins the dissection step of the procedure. The surgical hub 106, 206 may infer that the surgeon is dissecting to mobilize the patient's lung because it receives data from the RF generator or ultrasonic generator indicating that an energy instrument is being fired. The surgical hub 106, 206 may intersect the received data with the retrieval step of the surgical procedure to determine that the energy instrument fired at this point in the process (i.e., after the previously discussed surgical steps are completed) corresponds to the dissection step. In some cases, the energy instrument may be an energy tool mounted to a robotic arm of a robotic surgical system.

[0312] In step 10 5220, the surgical team proceeds with the ligation step of the procedure. The surgical hub 106, 206 may infer that the surgeon is ligating the artery and vein because it receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired. Similar to the previous step, the surgical hub 106, 206 may derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the retrieval step in the process. In some cases, the surgical instrument may be a surgical tool mounted to a robotic arm of a robotic surgical system.

[0313] Eleventh step 5222, perform the segmentectomy portion of the surgery. The surgical hub 106, 206 may infer that the surgeon is transecting soft tissue based on data from the surgical stapling and cutting instrument (including data from its bin). The bin data may correspond to, for example, the size or type of staples fired by the instrument. Since different types of staples are used for different types of tissue, the bin data may indicate the type of tissue being sutured and / or transected. In this case, the type of staple fired is for soft tissue (or other similar tissue types), which allows the surgical hub 106, 206 to infer that the segmentectomy portion of the surgery is in progress.

[0314] In the twelfth step 5224, a node dissection step is performed. The surgical hub 106, 206 can infer that the surgical team is dissecting the node and performing a leak test based on data received from the generator indicating that the RF or ultrasonic instrument is being fired. For this particular operation, the RF or ultrasonic instrument used after transecting the soft tissue corresponds to a node dissection step, which allows the surgical hub 106, 206 to make such inferences. It should be noted that the surgeon regularly switches back and forth between surgical suturing / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments according to the specific steps in the operation because different instruments are better suited for specific tasks. Therefore, a specific sequence in which suturing / cutting instruments and surgical energy instruments are used can indicate the steps of the operation being performed by the surgeon. In addition, in some cases, a robotic tool can be used for one or more steps in the surgical operation, and / or a handheld surgical instrument can be used for one or more steps in the surgical operation. One or more surgeons can, for example, alternate between a robotic tool and a handheld surgical instrument and / or can use the device simultaneously. When the twelfth step 5224 is completed, the incision is closed and the postoperative portion of the operation begins.

[0315] Thirteenth step 5226, reverse anesthesia to the patient. For example, the surgical hub 106, 206 may infer that the patient is waking up from anesthesia based on, for example, ventilator data (ie, the patient's breathing rate begins to increase).

[0316] Finally, the fourteenth step 5228 is for the medical staff to remove the various patient monitoring devices from the patient. Thus, the surgical hub 106, 206 can infer that the patient is being transferred to a recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices. As can be seen from the description of this exemplary surgery, the surgical hub 106, 206 can determine or infer when each step of a given surgical procedure occurs based on data received from various data sources that are communicatively coupled to the surgical hub 106, 206.

[0317] In various aspects, surgical device 214002 ( Figures 16 to 18 )、Display( Fig.16 )、wearable devices 214100、214102、214200、214202( Figures 19 to 21 ) and display 214400( Fig.22A and Fig. 22B ) is configured to be capable of being used in a hub environment such as a surgical hub 106 or 206 ( Figures 1 to 11 ) in a situationally aware manner, such as shown in timeline 5200. Situational awareness is further described in U.S. Provisional Patent Application Serial No. 62 / 659,900, filed on April 19, 2018, entitled "METHOD OF HUB COMMUNICATION," which is incorporated herein by reference in its entirety. In some cases, the operation of a robotic surgical system (including, for example, the various robotic surgical systems disclosed herein) can be controlled by the hub 106, 206 based on its situational awareness and / or feedback from its components and / or based on information from the cloud 104.

[0318] Control across surgical barriers

[0319] In various cases, the digital surgical device can be controlled through one or more surgical barriers. The surgical barrier includes a physical sterile barrier, an invisible sterile barrier, and a wall of the patient's body. In one aspect, the first operating room system located in the sterile area can be indirectly commanded and controlled by using a second operating room system, which can have a main operating mode and an auxiliary operating mode, and the second operating room system is designed to interact with the first operating room system to provide a command and control system. For example, in the main operating mode, the second operating room system can implement the main surgical function. In the auxiliary operating mode, the main surgical function can be disabled. In addition, in the auxiliary operating mode, the second operating room system can command and control the first operating room system. For example, the surgical instrument can be configured to be able to affect tissue in the first operating mode and command and control the imaging system and / or its display in the second operating mode.

[0320] In some cases, a clinician may wish to provide input to a remote surgical system and / or to a surgical hub communicatively coupled to the remote surgical system from within the sterile field. For example, a clinician holding a first operating room system (e.g., a surgical device) may wish to provide commands or input to another operating room system (e.g., an imaging or visualization system and / or its display) located outside the sterile field. In some cases, it may be desirable to utilize a first operating room system to communicate or interact with other operating room systems across a surgical barrier (such as the boundaries of a sterile field and / or a patient's body).

[0321] To facilitate such interaction, the first operating room system may include multiple operating modes, including a primary mode and an auxiliary mode. The first operating room system may switch between operating modes to selectively interact with the second operating room system. In this case, the clinician may interact with the second operating room system without handing over or putting down the first operating room system and / or without removing the first operating room system from the surgical site.

[0322] For example, a surgical system may include a first device and a second device, wherein the first device includes a first control circuit, and the second device is configured to enable a surgical function, wherein the second device includes a second control circuit in signal communication with the first control circuit, and wherein the second control circuit is configured to selectively switch the second device back and forth between an auxiliary operation mode and a primary operation mode, in which the second device is configured to indirectly control the first device, and in which the second device is configured to control the surgical function. Each control circuit may include a processor and a memory communicatively coupled to the processor, wherein the memory stores instructions executable by the processor to receive an input signal. In response to the input signal, the second control circuit may switch between the primary operation mode and the auxiliary operation mode. In the primary operation mode, the second control circuit may, for example, actuate the surgical function. When in the auxiliary operation mode, the second control circuit may, for example, control a display screen. In various cases, a non-tangible computer-readable medium may store computer-readable instructions that, when executed, cause the surgical device to receive an input signal. In response to the input signal, the second control circuit may switch between the primary operation mode and the auxiliary operation mode. In the primary operation mode, the second control circuit may, for example, actuate the surgical function. When in the auxiliary operating mode, the second control circuit can control the display screen, for example.

[0323] In one aspect, the surgical device may interface with the primary display to adjust and / or control the primary display. For example, the primary display may be controlled through a visual interface using auxiliary control functions of the surgical tool. In other words, the surgical device may be displayed on the primary display and may act as a cursor or indicator on the display to interact with the user interface of the display. In addition, the input to the surgical device may be input to the display and / or a surgical hub including the display. In this case, the surgical device may be an input device for an interactive technology, a user interface technology, or an input technology that uses a combination of hardware and software to allow a computer (or its control circuit) to perform a task. The output may be displayed on the primary display and / or transmitted to the surgical hub because the output generated by a mouse click and / or selection provides an input command that may be displayed or otherwise transmitted via a computer monitor, for example.

[0324] See also Figures 16 to 18 , the surgical system 214000 includes a surgical device 214002 and a display screen 214004. The display screen 214004 ( Fig.16 ) can be incorporated into an imaging system. For example, the imaging system can include an endoscope that houses an endoscopic camera and a display screen 214004 that is configured to display images obtained by the endoscopic camera. Endoscopes are further described in U.S. Patent Application No. 11 / 277,290 (now U.S. Patent Publication No. 2007 / 0225556), filed on March 23, 2006, entitled "DISPOSABLE ENDOSCOPE DEVICES," which is incorporated herein by reference in its entirety. Reusable endoscopes are also contemplated.

[0325] In various cases, the display screen 214004 may be a video monitor that is operably configured to display real-time feedback of images from the surgical site. The display screen 214004 may show real-time live video of the surgical site during a surgical procedure. In addition or alternatively, the display screen 214004 may be configured to display an augmented reality view of the surgical site. For example, the display screen 214004 may show hidden anatomical structures and / or hidden surgical devices. For example, such augmented reality views may be toggled on and off. The display screen 214004 also includes a graphical user interface. An operator may interact with the graphical user interface to provide input commands or controls to the display system and / or a surgical system communicatively coupled to the display system, as further described herein.

[0326] The surgical device 214002 is a handheld surgical instrument comprising a handle 214006, an elongated shaft 214008 extending distally from the handle 214006, and an end effector 214010 extending distally from the elongated shaft 214008. The end effector 214010 is configured to be able to affect tissue. In one case, the handheld surgical device may be an ultrasonic device. In this case, the end effector 214010 may include an ultrasonic knife. The clamping arm may be positioned opposite the ultrasonic knife to facilitate clamping tissue against the ultrasonic knife in various cases. In addition or alternatively, the end effector 214010 may include a tissue contact electrode configured to be able to deliver RF current to the tissue. In some cases, the end effector 214010 may include, for example, a reciprocating knife, a stapler, a clip applier, and / or a grasper. In some cases, the surgical device 214002 may include a housing releasably coupled to a robot arm. In this case, the clinician may control the surgical device 214002 at a surgeon console for a robotic surgical system.

[0327] The surgical device 214002 is configured to be switchable between a first mode and a second mode. The first mode may be an operational mode, in which the surgical device 214002 is configured to perform a surgical function. For example, in the first mode, the surgical device 214002 may be configured to apply vibration energy to tissue. The second mode may be a cursor mode or a control mode, in which the surgical device 214002 may be configured to provide input to the display screen 214004. The input may be configured to adjust the information displayed on the display screen 214004 and / or provide input to a connected surgical system such as a surgical hub, for example, the surgical hubs 106 and 206 ( Figures 1 to 11 )、Surgical Hub 7006( Fig.12 and Fig.13 ) and surgical hub 5104 ( Fig.14 ).

[0328] Main references Fig.17, the surgical device 214002 extends through the surgical barrier 214012 into the surgical site. The surgical barrier 214012 is the anatomical wall of the patient. The surgical device 214002 may also extend through the sterile field boundary forming another surgical barrier. At the surgical site, the imaging system is configured to be able to obtain a view of the surgical site and the surgical device 214002. For example, the distal portion of the surgical device 214002, i.e., the gripping tip / claw of the end effector 214010 and a portion of the shaft 214008 appear on the display screen 214004. In the first mode, the end effector 214010 can perform surgical functions, such as engaging, affecting and / or treating tissue. The first mode can be considered as the main mode because the end effector 214010 performs its main function as a surgical tool (e.g., a grasper can grasp tissue, an energy device can apply energy to tissue, a stapler can sew tissue, etc.). In the main mode, the end effector 214010 is configured to be able to directly affect tissue.

[0329] The end effector 214010 can be switched back and forth between a primary mode and a second mode (which can be referred to as an auxiliary mode). The selected mode can be displayed on the display screen 214004. For example, Fig.16 The display screen 214004 in the figure may display a cursor mode icon 214052 to indicate that the end effector 214010 acts as a cursor in the auxiliary mode. In the auxiliary mode, the end effector 214010 can be used to interact with the display screen 214004 and indirectly control the visualization system or the surgical system communicatively coupled thereto. The end effector 214010 displayed on the display screen 214004 can be used as a pointer or cursor of a graphical interface displayed on the display screen 214004. In this case, the clinician does not have to remove his hand from the surgical device 214002 or remove the surgical device 214002 from the patient's body to engage the display screen 214004 and provide input to the display screen 214004. In various cases, the camera of the imaging device (e.g., a laparoscopic camera) can be configured to be able to track the movement of the surgical device at the surgical site. For example, the camera can scan or otherwise adjust its field of view to follow one or more surgical devices (or portions thereof) around the surgical site. In various cases, the clinician can select which surgical device and / or portion thereof the camera tracks. In this case, the camera can track the end effector 214010, which can ensure that the end effector 214010 is displayed on the display screen 214004 when the surgical device 214002 is in the auxiliary mode.

[0330] In the auxiliary mode, the surgical device 214002 can be used as an input device, such as a computer mouse or joystick, for example, to move a cursor around an interface on the display screen 214004 to manipulate functions displayed on the display screen 214004. When the display portion of the device is used as a cursor, the device prompt (i.e., the "cursor") can press buttons on the display screen 214004, and drag and drop display items, circle and / or highlight a portion of the video (e.g., point out a tumor or unique anatomical feature) for later reference. For example, the device prompt can select the home screen icon 214040 to return to the home screen, select the play icon 214042 to play a video of the surgical procedure obtained by the endoscope, select the pause icon 214044 to pause the video, select the playback icon 214046 to re-watch or replay a portion of the video, select the record icon 214048 to record a new portion of the surgical procedure, and / or select the settings icon 214050 to adjust the settings and / or view on the display screen 214004. In one aspect, activating an auxiliary mode (or cursor control mode) can change the way a tool works in this mode. For example, a button, trigger, and / or other actuator can have different functions in different modes.

[0331] The operating mode of the surgical device 214002 can be selected by the clinician. For example, the clinician can selectively switch back and forth or switch the surgical device 214002 between the primary mode and the auxiliary mode. In one case, the auxiliary mode can be activated by a voice command. In another case, the clinician's tactile action can activate the auxiliary mode. Fig.18 , the surgical device 214002 includes a manual switch 214018 that can enable a clinician to switch between operating modes. A first position of the manual switch 214018 can correspond to a primary mode, and a second position of the manual switch 214018 can correspond to an auxiliary mode. The clinician can move the manual switch 214018 to switch the surgical device 214002 back and forth between operating modes.

[0332] In one example, surgical device 214002 is an ultrasonic surgical instrument, such as the HARMONIC Scissors. Such ultrasonic surgical instruments may include multiple input actuators, such as a maximum power button 214020 and a minimum power button 214022. The maximum power button 214020 can generate ultrasonic energy at a first energy level or in a first range in the main mode, and the minimum power button 214022 can generate ultrasonic energy at a second energy level or in a second range in the main mode. The second energy level or second range may be less than the first energy level or first range. In various cases, the buttons 214020, 214022 may define a position range corresponding to different levels and / or may detect the force of the operator and adjust the energy level accordingly. For example, at least one of the buttons 214020, 214022 may define a rotating element to scroll between levels and / or selections, such as similar to a wheel on a computer mouse.

[0333] In the auxiliary mode, the ultrasonic sealing and cutting functions can be disabled, and the maximum power button 214020 and the minimum power button 2104022 on the surgical device 214002 can function like the buttons of a computer mouse. The clinician can point the prompt of the surgical device 214002 (as displayed on the display screen 214004) to something displayed on the display screen 214004, and the buttons can then interact with the display screen 214004, as described above. For example, the dual button input feature of the surgical device 214002 can be intuitive to clinicians who are familiar with dual button computer mice. In various cases, the dual buttons can be used to select icons, drag and drop icons, and / or adjust and interact with various features of the display screen 214004, as further described herein.

[0334] In various embodiments, display screen 214004 is communicatively coupled to a surgical hub, such as surgical hub 106 or 206 ( Figures 1 to 11 )、Surgical Hub 7006( Fig.12 and Fig.13 ) or Surgical Hub 5104 ( Fig.14 In one embodiment, the visualization system 108 (see Figure 2 ) can be the display screen 214004. In this case, the visualization system 108 and its display screen 214004 can be coupled to the imaging module 138 (see Figure 3). In addition, the situational awareness of the surgical hub 106 can be configured to enable intelligent adjustments and / or recommendations to the clinician during the surgical procedure. For example, the situational awareness module of the surgical hub 106 can suggest that the clinician confirm complete tissue seal, suture alignment, and / or tumor removal, and in response to the recommendation, the clinician can switch the surgical device 214002 from the main mode to the auxiliary mode to interact with the display screen 214004. For example, the clinician can interact with the display screen 214004, such as by zooming in on the tissue seal, expanding the view on the display screen 214004 to check the tissue seal, and / or selecting a measurement tool from the graphical interface to measure the length of the tissue seal and / or another anatomical structure / landmark distance. The clinician can respond to the prompts / suggestions of the surgical hub without removing the surgical device 214002 from the surgical site or removing his or her hands from the surgical device 214002.

[0335] In various cases, the clinician can interact with the display screen 214004 to control the surgical hub and / or provide input to the surgical hub. In the case where the display screen 214004 is positioned outside the sterile field, for example Figure 2 The main display 119 in the sterile area, the clinician can control the display screen 214004 outside the sterile area through various surgical barriers (including the sterile area boundary and the patient's body). In other words, the clinician can use the first operating room system to indirectly control and / or command the second operating room system outside the sterile area completely in the sterile area.

[0336] In various aspects, a wearable device, i.e., a "wearable," may be configured to facilitate interaction with one or more communicatively coupled devices. For example, a wearable device of a clinician within a sterile field may be used to interact with a surgical system outside the sterile field. The wearable device may be an interactive device configured to enable interaction with a remote system. In various cases, the wearable device may identify the wearer, i.e., the clinician wearing the device, and may identify surgical devices within a range of locations around the wearable device. Such a wearable device may determine, for example, whether and how the clinician is holding a particular surgical device. Fig. 20 FIG. 2 shows a wrist-worn wearable 214100, Fig.21 214102 is shown in FIG. For example, the wrist-worn wearable 214100 may be attached to the clinician's wrist "W", such as a watch or bracelet. For example, the finger-worn wearable 214102 may be attached to the clinician's thumb "T" or a finger "F" of the clinician's hand, such as a ring.

[0337] Wearables 214100, 214102 each include a communication module 214104 that facilitates communication between the wearables 214100, 214102 and another surgical system. The communication module 214104 can enable RFID tags and / or near field communications. The communication module 214104 can transmit a wireless signal 214106 indicating a command or control from a clinician. For example, wearables 214100 and 214102 can include a graphical user interface and / or a touch screen. The clinician can engage the touch screen or otherwise provide input to the graphical user interface to enable adjustment of another surgical device. In one embodiment, the wearable 214100 and / or 214102 can communicate signals with a visualization system and / or its display screen. For example, the clinician can engage arrows 214110 on the user interface to pan the view on the main display of the surgical room, adjust the zoom feature 214112 ( Fig. 20 ) or 214113( Fig.21 ) to zoom in or out the view on the primary display, and / or select one or more icons 214114, 214116 on the user interface to adjust the view and / or data displayed on the display screen. Alternative graphics and input features are contemplated.

[0338] In one aspect, the wearable device can assist the clinician in interacting with a display, such as a primary display, such as primary display 119 ( Figure 2 ). For example, the wearable device may allow input from the surgeon to select, advance, resize, and gesture relative to a graphical user interface on the wearable device to adjust or control the primary display. For example, inputting a zoom operation on the wearable device may magnify a portion of the video feedback on the primary display. In one instance, the wearable device may include a simplified and / or minimized version of the information on the primary display, similar to a mobile device page that displays a simplified and / or minimized version of the available information on a desktop site. The wearable device may include a graphical user interface and a touch screen such that input to the graphical user interface on the touch screen may be transmitted to the primary display. In one such implementation, the clinician may interact with a sealed capacitive interactive surface on the wearable device, which may be worn like a wristband watch, for example using a capacitively infused latex glove.

[0339] In various cases, image and / or object recognition techniques may be used to track wearable devices. For example, various machine vision, image processing, object recognition, and optical tracking techniques may be used to process image and / or video data to track features, characteristics, actions, and movements of wearable devices. For example, a variety of image and / or object recognition techniques, including appearance- and feature-based techniques, may be used to identify wearable devices from images captured by one or more cameras in a surgical room. For example, an edge detection algorithm (e.g., a Canny edge detector algorithm) may be used to process captured images to generate contours of various objects within each image. The algorithm may then compare a template of a target object (e.g., a target wearable device) with an image containing contour objects to determine whether any of the target objects is located within the image. As another example, the algorithm may extract features from the captured image. The extracted features are then fed to a machine learning model (e.g., an artificial neural network or a support vector machine) trained via supervised or unsupervised learning techniques to associate feature vectors with targets. These features may include edges (e.g., extracted via a Canny edge detector algorithm), curvature, corners (e.g., extracted via a Harris & Stephens corner detector algorithm), and the like. Object recognition and tracking are further described in the following patent applications: U.S. patent application Ser. No. __________, filed concurrently, entitled “USAGE AND TECHNIQUE ANALYSIS OF SURGEON / STAFF PERFORMANCE AGAINST A BASELINE TO OPTIMIZE DEVICE UTILIZATION AND PERFORMANCE FOR BOTH CURRENT AND FUTURE PROCEDURES,” attorney docket No. END9018USNP2 / 180517-2; and U.S. patent application Ser. No. __________, filed concurrently, entitled “IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE INUSE,” attorney docket No. END9018USNP3 / 180517-3, the entirety of each of which is incorporated herein by reference.

[0340] In some cases, the wearable device may include an array of magnets that can be detected by sensors in the surgical room. Based on the position of the wearable device detected by the sensor, the surgical system can determine the movement of the wearable device. For example, the movement may correspond to a posture of the clinician. In this case, the surgical system can determine one or more postures of the clinician, and in various cases, such postures can be transmitted to another surgical system, such as an imaging system. For example, the posture may correspond to an input command to a display screen of the imaging system. The magnetic sensing array is further described in U.S. patent application No. __________, entitled "IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROLOF A SURGICAL DEVICE IN USE," filed at the same time, with attorney docket number END9018USNP3 / 180517-3, which is incorporated herein by reference in its entirety.

[0341] As another example, such as when, for example, a handheld surgical instrument is picked up and / or handed over to a clinician, the wearable device can track the surgical device by pairing with the surgical device during the handover. Fig.19 , clinician C wears two wearable devices, a first wearable device 214200 on the clinician's right wrist RW, and a second wearable device 214202 on the clinician's left wrist LW. The wearable devices 214200, 214202 may be configured to be able to determine which surgical device is positioned in which hand of the clinician. For example, when the second surgical device 214212 is handed over to the clinician, the first wearable device 214200 may be paired with the first surgical device 214210 on the clinician's right hand, and the second wearable device 214202 may be paired with the second surgical device 214212. In one specific implementation, the wearable device 214200 and / or 214202 may include a built-in RFID tag or near field communication device that allows the handle in the clinician's hand to identify which clinician is holding the surgical device, thereby allowing the surgical device to be automatically paired or re-paired to the clinician. For example, clinician-desired settings and / or most commonly used features / adjustments specific to that particular surgical device may appear on and / or be highlighted on the screen of the wearable device. Fig.19A wrist-worn device is shown in the figure, but the reader will recognize that alternative devices are contemplated, such as a ring and / or a glove. In addition, in some cases, the clinician may only be able to wear the wearable device on one wrist, such as their dominant wrist, for example. In various cases, the wearable device can determine the orientation of the surgical tool relative to the wearable device, thereby determining the orientation relative to the clinician's hand. In this case, the surgical system can adjust the display and controls based on the detected orientation of the surgical device on the clinician's hand.

[0342] For example, a first arrangement of controls and / or selections may be available to a right-handed clinician, and a second, different arrangement of controls and / or selections may be available to a left-handed clinician. Additionally or alternatively, the arrangement of controls and / or the availability of certain controls may depend on whether the surgical device is positioned in the clinician's dominant or non-dominant hand and / or based on situational awareness from the surgical hub. In various cases, the wearable device may, for example, identify the clinician, thereby including clinician-customized settings, including, for example, identifying the clinician's dominant hand.

[0343] In various cases, the control of the surgical device may be shared by different control devices and / or the control of the surgical device may be switched between multiple control devices. For example, the surgical device may include an autonomous control mode in which the control input is provided by the device itself. For example, a clinician may engage an actuator (e.g., a button, a switch, a toggle, a trigger, etc.) on the surgical device to actuate a surgical function (e.g., energy activation, clamping, firing, etc.) of the surgical device. Handheld surgical tools and robotic surgical tools may, for example, operate in an autonomous control mode. For robotic surgical tools, the input controls may be on the surgeon's console. For handheld surgical tools, the input controls may, for example, be on the handle.

[0344] In some cases, the surgical device may be controlled and / or dominated by another surgical device, which may selectively send a control input to the surgical device. In this case, the surgical device may be referred to as a "controlled surgical device", and the other surgical device may be referred to as a "controlling surgical device". In one aspect, a mobile device such as a smart phone or a tablet computer with a wireless communication feature may be a controlling surgical device, which is selectively configured to provide a control input to the controlled surgical device. Such a mobile device may be positioned in a sterile area. In addition or alternatively, a wearable device may be a controlling surgical device, which is configured to provide a control input to the controlled surgical device, as further described herein. In other cases, a controlling surgical device that may be paired and / or communicatively coupled to the controlled surgical device via a surgical hub may be configured to provide an input control to the controlled surgical device. In other cases, a display screen may be a controlling surgical device, which is configured to provide an input control to the controlled surgical device, and / or a controlling surgical device may interact with a visualization system (e.g., as a cursor on a display screen) to provide input control to the visualization system and / or another connected surgical device (i.e., the controlled surgical device). In the foregoing examples, the controlled surgical device may maintain at least some degree of autonomous control, while the controlling surgical device also selectively exercises varying degrees of control over the controlled surgical instrument. In other cases, input controls of the controlled surgical device may be disabled when the control of the controlling surgical device is enabled and / or activated. In various cases, multiple surgical devices (including the controlled surgical device itself in some cases) may simultaneously share control of the controlled surgical device. The reader will recognize that various interactions of control interactions are contemplated where a surgical hub couples multiple surgical devices together to form a collaborative surgical system.

[0345] In situations where multiple surgical devices share control and / or alternate functionality between control functions, a clinician may wish to know which controlling surgical devices have control of a controlled surgical device at a particular time during a surgical procedure. For example, a surgical system may provide a clinician with tactile, auditory, and / or visual indications of control modes.

[0346] In one aspect, the surgical system may provide various tactile, audio, and / or visual prompts to the clinician, for example, via a user interface. The surgical system may highlight, emphasize, or otherwise draw attention to display prompts on the user interface. In addition or alternatively, the surgical system may nest or overlay data and key information. For example, certain information may be provided with an augmented reality view on the user interface and / or through live feedback of images, videos, or other real-time data obtained by the surgical system. On the other hand, the surgical system may provide enhanced indications of control functions and / or their limitations. For example, when the control of the surgical device is disabled or otherwise infeasible, the surgical device, controls, or display may vibrate to convey that the control input was not accepted.

[0347] For example, if situational awareness indicates that a gross motion step is being performed during a surgical operation, a clinician in the sterile area can be enabled to manipulate the position of a controlled surgical instrument from within the sterile area via a control surgical device, such as by using a mobile tablet or wearable device located within the sterile area. However, if situational awareness indicates that a fine motion step is being performed during a surgical operation, the control function of the control surgical device can be disabled so that only a clinician operating the surgical device (such as a clinician at a surgeon's console) can provide input control to the controlled surgical device. In this case, the control surgical device can provide tactile, auditory and / or visual notifications to the clinician in the sterile area to indicate that such control features are disabled. For example, the control surgical device can simply enter a "sleep" mode so that the sterile area clinician cannot provide input. In some cases, the control surgical device can, for example, provide voice notifications and / or prompts to convey that the desired control function is not feasible. In addition or alternatively, when a clinician in the sterile area attempts to provide an infeasible control input, the control surgical device can vibrate or otherwise provide tactile feedback.

[0348] As another example, in various cases, the autonomous control of a surgical device may override the control of another surgical device over the surgical device. For example, if a clinician at a surgeon's console is actively controlling a robotic tool, auxiliary control, such as control input provided by a clinician in a sterile field via a control surgical device, may be ignored. However, when a clinician at a surgeon's console stops actively controlling the robotic tool, control input from the control surgical device in the sterile field may control the robotic tool. When the robotic tool is controlled by a control surgical device in the sterile field, a user interface at the robotic console may communicate that control is being shared with the control surgical device using tactile, auditory, and / or visual cues. Similarly, when a robotic tool is controlled by a clinician at a surgeon's console, a user interface on the control surgical tool may indicate that its control function is disabled.

[0349] In various cases, controls for a surgical device may be configured to interface with one or more displays to communicate interactions between the various controls. For example, a controlling surgical device may be configured to provide auxiliary control functions to a controlled surgical device. Additionally, one or more indications may indicate or otherwise communicate paired control functions between a controlled surgical device and a controlling surgical device. Such indications may be provided on one or more displays, such as non-sterile displays 107 and 109, primary display 119, hub display 135, and / or a display in surgeon console 118, such as, for example, Figure 2 and Figure 3As shown. In one aspect, a clinician can use a touch screen on a surgical device to interact with the main surgical hub display. For example, a surgical device in a sterile field may include a capacitive touch screen display. Surgical devices such as handheld surgical instruments, robotic tools, wearables, and display screens may include a touch screen within the sterile field. In various cases, a clinician in a sterile field can slide, scroll, and / or otherwise configure the main surgical hub display from a touch screen within the surgical field. For example, a clinician can scan, zoom, or otherwise adjust the main display of the surgical hub by scanning, zooming, or otherwise adjusting the view on the touch screen within the sterile field.

[0350] In various cases, the auxiliary device may include controls that can only be paired with other devices to link command and control functions. For example, the auxiliary device may be a controller for controlling one or more other surgical devices. In various cases, the auxiliary device or control surgical device may be a screen located in the sterile field. For example, the auxiliary device may be a mobile device such as a tablet or mobile phone that includes a screen. The screen may be, for example, a touch screen that is configured to be able to receive control inputs from a clinician in the sterile field. In addition or alternatively, the display may include embedded LEDs that highlight the control function in a manner that pairs or connects the control function to a specific controlled surgical device.

[0351] See also Fig.22A and Fig. 22B , showing display 214400. Display 214400 in Fig.22A The first plurality of information 214402 is shown in Fig. 22B A second plurality of information 214404 is shown in FIG. The user can interact with the display 214400 to Fig.22A and Fig. 22B For example, the display 214400 includes a capacitive screen. In various cases, the clinician can interact with the interface portion of the display 214400 by touching it. The display 214400 is configured to be able to convey different types of information to the clinician. For example, the display 214400 can convey patient-specific information by selecting a PATIENT icon 214410, convey procedure-specific information by selecting a PROCEDURE icon 214412, and convey device-specific information by selecting a DEVICE icon 214414. Fig.22A and Fig. 22B4, the DEVICE icon 214414 is selected, as indicated by the highlight around the DEVICE icon 214414. In other cases, the selected icon 214410, 214412, 214414 may be communicated in another manner, such as showing the selected icon 214410, 214412, 214414 in a different size, a different position, a different color, a different style, and / or with another identifier such as an arrow or a symbol associated therewith. Fig.22A and Fig. 22B 214400. In various embodiments, the display 214400 may selectively provide input commands to one or more surgical devices during a surgical procedure.

[0352] The surgical devices listed are a combined energy surgical device indicated by a first icon 214420, a surgical stapler indicated by a second icon 214422, and a suction / irrigation device indicated by a third icon 214424. Icons 214420, 214422, 214424 are Fig.22A and Fig. 22B ; however, in other cases, icons 214420, 214422, and 214424 may include graphics and / or may be, for example, symbols, such as symbolic representations of surgical devices. In some cases, fewer than three or more surgical devices may be listed. In some cases, only controllable surgical devices may be listed. In other cases, all surgical devices used during a surgical procedure may be listed; however, the clinician may only be able to select controllable surgical devices from the list. Alternative surgical devices are contemplated (e.g., ultrasonic devices, electrosurgical devices, clip applicators, graspers, knives, etc.).

[0353] exist Fig.22A The icon 214420 corresponding to the combined energy surgical device is selected in Fig. 22B The icon 214422 corresponding to the stitcher is selected, as indicated by the highlight around the corresponding icon 214420, 214422. In other cases, the selected icon 214420, 214422, 214424 may be communicated in another manner, such as showing the selected icon 214420, 214422, 214424 in a different size, a different position, a different color, a different style, and / or with another identifier such as an arrow or a symbol associated therewith. Fig.22AAn icon 214420 corresponding to a combined energy surgical device is selected in the display 214400, so the available input commands correspond to commands for the combined energy surgical device. For example, the clinician can select a mode from the following mode list 214430: ultrasonic, monopolar, bipolar, and hybrid. Alternative modes are also conceivable. The clinician can select a mode to control the energy modality applied by the combined energy surgical device. In other cases, additional adjustments such as power level, duration, frequency, etc. can be controlled from the display 214400. In various cases, user inputs to the display 214400 can be transmitted to the surgical hub via one or more communication protocols described herein. Similarly, the surgical hub can relay user inputs to the combined energy surgical device.

[0354] See now Fig. 22B , because the icon 214422 corresponding to the surgical stapler is selected, the available input commands correspond to the commands for the surgical stapler. For example, the clinician can select the surgical function from a plurality of modes 214432 such as the clamping mode and the firing mode. In various cases, the mode 214432 may include another subcategory or adjustment that can be selected from the menu. For example, the clamping and / or firing can be performed manually or automatically, and / or the speed can be selected from a variety of speeds, such as slow, medium and fast. It is also conceivable to select an alternative mode. The clinician can select a mode to control the surgical function of the surgical stapler. In various cases, the user input of the display 214400 can be transmitted to the surgical hub by one or more communication protocols described herein. Similarly, the surgical hub can relay the user input to the surgical stapler.

[0355] In various cases, the pairing between the display 214400 and the surgical device may be communicated to the clinician. For example, the display 214400 may include at least one embedded LED that may illuminate a color that matches the color of the controlled surgical device and / or a color identifier on the controlled surgical device. In one aspect, the controlled surgical device may include, for example, a similar LED color identifier. The embedded LED on the display 214400 may illuminate red, and the red LED on the paired surgical device may illuminate to indicate the pairing. In various cases, the display 214400 may be configured to be able to control multiple surgical devices, and in such a case, multiple colors may be displayed on the display 214400, and corresponding colors may be provided on corresponding identifiers throughout the surgical room. For example, when the clinician selects the first icon 214420 ( Fig.22A ), a portion of the display 214400 (e.g., the first icon 214420) may light up a particular color and the LED on the combined energy surgical device may light up the same color. Similarly, when the clinician selects the second icon 214422 ( Fig. 22B), a portion of the display 214400 (e.g., the second icon 214422) may light up a particular color, and the LED on the surgical stapler may light up the same color. Different colors may be assigned to different surgical devices. In various cases, the icons 214420, 214422, 214424 on the display may light up different colors and / or the icons may highlight in different colors. For example, the shape around the selected icon 214420, 214422, 214424 may be a particular color corresponding to the indicator on the paired surgical device.

[0356] The identifier on the controlled surgical device (e.g., a combined energy surgical device, a surgical stapler, or a suction / irrigation device) can be displayed in different ways or styles to convey different control states. For example, when the control of the display 214400 is disabled, the identifier may display a first pattern or style (e.g., the identifier may be an unlit LED). When the control of the auxiliary device is enabled, the identifier may display a second pattern or style (e.g., the identifier may be a lit LED), and / or when the auxiliary device is paired with the controlled surgical device, the identifier may display a third pattern or style (e.g., the identifier may be a flashing LED). As described above, the color of the LED may correspond to the color on the display 214400.

[0357] In various cases, one control interface can be used to activate a surgical function and another control interface can be used to sequence through the modes of the instrument. For example, a single trigger on a surgical device can implement different surgical functions, and the specific surgical function can be determined by the control interface on the display 214400. More specifically, with respect to Fig.22A , a trigger or actuator on a combined energy surgical device may apply ultrasonic energy to tissue; however, if the clinician selects another mode, the same trigger or actuator will apply a different energy modality to the tissue. In various cases, the energy modality and / or power level may be controlled via display 214400. See now Fig. 22B , the trigger or actuator on the surgical stapler can be configured to clamp when the clamp function is selected on the display 214400, and the trigger can be configured to fire when the fire function is selected on the display 214400. Additional adjustments (e.g., manual or automatic, clamp speed, fire speed, etc.) can be adjusted on the display 214400.

[0358] In other cases, multiple control devices may have actuation capabilities for the same surgical device. For example, display 214400 may include actuation controls for actuating surgical functions on a surgical device. In some cases, the power level of each actuation may vary between different control devices.

[0359] In one aspect, the control of the auxiliary function may allow fine control from one controller and overall control from another controller. The auxiliary function may include, for example, joint movement of the end effector and rotation of the distal head. In this case, when overall control is required, the integrated or built-in device control (e.g., autonomous control on the surgical device) can control the function. When fine control is required, the auxiliary controller (e.g., display 214400) can control the function. For example, when the clinician is using another aspect of the surgical device, such as holding the trigger, the fine control function can still be achieved by the auxiliary controller paired with the surgical device to produce fine movement of the auxiliary function. In some cases, when another function (such as energy actuation, clamping or emission) is in operation, certain functions (such as joint movement and rotation of the distal head) can be locked. However, when the auxiliary controller for the auxiliary function is actuated by another clinician or the other hand of the same clinician, fine adjustment can be allowed even though the clinician is using another control to operate another function. In this operating mode, the auxiliary function may have a limited operating envelope to ensure that the force applied and / or the rate of its operation is limited to produce the desired fine control or fine precision. Additionally or alternatively, it may override certain threshold limits to some extent because it is directly controlled.

[0360] Example

[0361] Various aspects of the subject matter described herein are set forth in the following numbered examples:

[0362] Embodiment 1 - A surgical system includes a first device and a second device, wherein the first device includes a first control circuit and the second device is configured to perform a surgical function. The second device includes a second control circuit that communicates signals with the first control circuit. The second control circuit is configured to selectively switch the second device back and forth between an auxiliary operating mode and a primary operating mode, wherein the second device is configured to control the first device in the auxiliary operating mode and the second device is configured to control the surgical function in the primary operating mode.

[0363] Example 2 - The surgical system of Example 1, wherein the first device comprises a display, wherein the second device comprises an end effector positioned within the sterile field, and wherein the end effector is visible on the display.

[0364] Example 3 - The surgical system of Example 1 or 2, wherein the auxiliary operating mode includes a cursor mode, and wherein the primary operating mode includes a tissue treatment mode.

[0365] Example 4 - The surgical system of any one of Examples 1 to 3, wherein the second device includes a handle that includes an input switch capable of moving between a first position and a second position, and wherein the first position corresponds to a primary operating mode and the second position corresponds to an auxiliary operating mode.

[0366] Example 5 - The surgical system of any of Examples 1 to 4, wherein the second control circuit is configured to be capable of switching back and forth between the primary operating mode and the secondary operating mode in response to a clinician's voice command.

[0367] Example 6 - The surgical system of Example 3, wherein the end effector is configured to be capable of dragging and dropping icons on the display in cursor mode.

[0368] Example 7 - The surgical system of Example 3 or 6, wherein the end effector is configured to be capable of selecting anatomical features on the display in cursor mode.

[0369] Example 8 - The surgical system of any one of Examples 1 to 7, wherein the second device includes an ultrasonic instrument configured to apply ultrasonic vibrations to tissue, wherein the ultrasonic instrument includes a first actuation button and a second actuation button, wherein, in a primary operating mode, the first actuation button is configured to actuate a first energy level and the second actuation button is configured to actuate a second energy level, and wherein, in an auxiliary operating mode, the first actuation button includes a first cursor button and the second actuation button includes a second cursor button.

[0370] Example 9 - A surgical system, the surgical system comprising an imaging system, the imaging system comprising a camera and a display screen. The surgical system also comprises a surgical device, the surgical device configured to perform a surgical function. The surgical device comprises a control circuit, the control circuit comprising a processor and a memory communicatively coupled to the processor, the memory storing instructions executable by the processor to receive an input signal, switch between a first operating mode and a second operating mode in response to the input signal, actuate the surgical function in the first operating mode, and control the display screen in the second operating mode.

[0371] Example 10 - The surgical system of Example 9, wherein the surgical device is configured to be able to control the display screen through the surgical barrier.

[0372] Example 11 - The surgical system of Example 9 or 10, wherein the display screen comprises a video monitor in an operating room, and wherein the surgical device comprises a laparoscopic device comprising an end effector positioned within a patient in the operating room.

[0373] Example 12 - The surgical system of Example 11, wherein the surgical device includes an end effector and wherein the camera is configured to be able to track the end effector within the patient's body.

[0374] Example 13 - The surgical system of Example 11 or 12, wherein, in the second operating mode, the end effector is configured to be able to interact as a cursor with one or more icons on the video monitor.

[0375] Example 14 - The surgical system of any of Examples 11 to 13, wherein in the second operating mode, the end effector is configured to be capable of interacting as a cursor with video feedback on the video monitor.

[0376] Example 15 - The surgical system of any one of Examples 9 to 14, wherein the surgical device includes a handle that includes an input switch capable of moving between a first position and a second position, and wherein the first position corresponds to a first operating mode and the second position corresponds to a second operating mode.

[0377] Example 16 - The surgical system of any one of Examples 9 to 14, wherein the control circuit is configured to be able to switch back and forth between the first operating mode and the second operating mode in response to a clinician's voice command.

[0378] Example 17 - A non-transitory computer-readable medium storing computer-readable instructions that, when executed, cause a surgical system to: receive an input signal, switch between a first operating mode and a second operating mode in response to the input signal, actuate a surgical function in the first operating mode, and interact with a display screen through a surgical barrier in the second operating mode.

[0379] Example 18 - The non-transitory computer readable medium of Example 17, wherein the surgical system is configured to be able to interact with the display screen through the surgical barrier by clicking on icons on the display screen.

[0380] Example 19 - The non-transitory computer readable medium of Example 17 or 18, wherein the surgical system is configured to be able to interact with the display screen through the surgical barrier by dragging and dropping icons on the display screen.

[0381] Example 20 - The non-transitory computer readable medium of any one of Examples 17 to 19, wherein the surgical system is configured to be able to interact with the display screen through the surgical barrier by selecting a portion of the video.

[0382] Although multiple forms have been illustrated and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such details. Without departing from the scope of the present disclosure, many modifications, changes, alterations, substitutions, combinations and equivalents to these forms can be realized, and those skilled in the art will think of many modifications, changes, alterations, substitutions, combinations and equivalents to these forms. In addition, alternatively, the structure of each element associated with the described form can be described as a device for providing the function performed by the element. In addition, in the case of disclosing materials for certain parts, other materials can also be used. Therefore, it should be understood that the above-mentioned specific embodiments and the appended claims are intended to cover all such modifications, combinations and variants within the scope of the forms disclosed by the present invention. The appended claims are intended to cover all such modifications, changes, alterations, substitutions, modifications and equivalents.

[0383] The above specific embodiments have been described in various forms of devices and / or methods by using block diagrams, flow charts and / or examples. As long as such block diagrams, flow charts and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation in such block diagrams, flow charts and / or examples can be implemented individually and / or collectively by a variety of hardware, software, firmware or any combination thereof. Those skilled in the art will recognize that some aspects of the forms disclosed herein can be implemented equivalently in integrated circuits as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as any combination thereof in whole or in part, and according to the present invention, designing electronic circuits and / or writing software and / or hardware codes will be within the technical scope of those skilled in the art. In addition, those skilled in the art will recognize that the mechanisms of the subject matter described herein can be distributed as one or more program products in various forms, and the exemplary forms of the subject matter described herein are applicable regardless of the specific type of signal-bearing medium used to actually distribute.

[0384] Instructions for programming logic to perform various disclosed aspects may be stored in a memory within the system, such as a dynamic random access memory (DRAM), a cache, a flash memory, or other memory. In addition, the instructions may be distributed via a network or through other computer-readable media. Therefore, a machine-readable medium may include a mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but is not limited to a floppy disk, an optical disk, an optical disk, a read-only memory (CD-ROM), a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible, machine-readable storage device used when transmitting information on the Internet via an electrical signal, an optical signal, an acoustic signal, or other form of propagation signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.). Therefore, a non-transitory computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0385] As used in any aspect of this document, the term "control circuitry" may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more separate instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), a field programmable gate array (FPGA)), state machine circuitry, firmware that stores instructions executed by the programmable circuitry, and any combination thereof. The control circuitry may be implemented collectively or individually as circuitry that forms part of a larger system, such as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smart phone, etc. Thus, as used herein, "control circuitry" includes, but is not limited to, electronic circuitry having at least one discrete circuit, electronic circuitry having at least one integrated circuit, electronic circuitry having at least one application specific integrated circuit, electronic circuitry forming a general purpose computing device constructed by a computer program (e.g., a general purpose computer constructed by a computer program that at least partially implements the methods and / or devices described herein, or a microprocessor constructed by a computer program that at least partially implements the methods and / or devices described herein), electronic circuitry forming a memory device (e.g., forming a random access memory), and / or electronic circuitry forming a communication device (e.g., a modem, a communication switch, or an optoelectronic device). Those skilled in the art will recognize that the subject matter described herein may be implemented in an analog or digital manner, or some combination thereof.

[0386] As used in any aspect of this document, the term "logic" may refer to an application, software, firmware, and / or circuitry configured to perform any of the foregoing operations. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, instruction sets, and / or data hard-coded (e.g., non-volatile) in a memory device.

[0387] As used in any aspect of this document, the terms "device," "system," "module" and the like may refer to a computer-related entity, hardware, a combination of hardware and software, software, or software in execution.

[0388] As used in any aspect herein, an "algorithm" refers to a self-consistent sequence of steps leading to a desired result, wherein the "steps" refer to manipulations of physical quantities and / or logical states which may, but need not, take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. References to such signals are often made to bits, values, elements, symbols, characters, terms, numbers, and the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to such quantities and / or states.

[0389] The network may include a packet switching network. The communication devices may be able to communicate with each other using the selected packet switching network communication protocol. An exemplary communication protocol may include an Ethernet communication protocol that allows communication using a transmission control protocol / Internet protocol (TCP / IP). The Ethernet protocol may comply with or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" issued by the Institute of Electrical and Electronics Engineers (IEEE) in December 2008 and / or a higher version of this standard. Alternatively or additionally, the communication devices may be able to communicate with each other using an X.25 communication protocol. The X.25 communication protocol may comply with or comply with the standards promulgated by the International Telecommunication Union Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, the communication devices may be able to communicate with each other using a frame relay communication protocol. The frame relay communication protocol may comply with or comply with the standards issued by the International Telephone and Telephone Consultation Committee (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be able to communicate with each other using an asynchronous transfer mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard entitled "ATM-MPLS Network Interworking 2.0" released by the ATM Forum in August 2001 and / or a later version of the standard. Of course, different and / or later developed connection-oriented network communication protocols are also contemplated herein.

[0390] Unless otherwise expressly indicated in the above disclosure, it is understood that discussions using terms such as "processing," "computing," "calculating," "determining," and "displaying" in the above disclosure refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms it into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0391] One or more components may be referred to herein as "configured to be able to", "configurable to be able to", "operable / operably", "suitable / adaptable to", "able to", "conformable / conformable to", etc. Those skilled in the art will recognize that, unless the context indicates otherwise, "configured to be able to" may generally cover components in an active state and / or components in an inactive state and / or components in a standby state.

[0392] The terms "proximal" and "distal" are used herein relative to a clinician manipulating the handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion positioned away from the clinician. It should also be understood that for brevity and clarity, spatial terms such as "vertical," "horizontal," "upper," and "lower" may be used herein in conjunction with the accompanying drawings. However, surgical instruments are used in many orientations and positions, and these terms are not restrictive and / or absolute.

[0393] Those skilled in the art will recognize that, in general, the terms used herein, and in particular in the appended claims (e.g., the body of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be interpreted as "including but not limited to", the term "having" should be interpreted as "having at least", the term "comprising" should be interpreted as "including but not limited to", etc.). Those skilled in the art will also understand that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such a recitation, no such intent is present. For example, to aid understanding, the following appended claims may contain use of the introductory phrases "at least one" and "one or more" to introduce claims. However, the use of such phrases should not be construed as implying that the introduction of a claim recitation by the indefinite article "a" or "an" limits any particular claim containing such introduced claim recitation to claims containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should generally be interpreted as meaning "at least one" or "one or more"); this also applies to the use of definite articles used to introduce claim recitations.

[0394] In addition, even if a specific number of claim recitations is explicitly recited, those skilled in the art will recognize that such recitation should generally be interpreted to mean at least the recited number (e.g., a bare recitation of "two recitations" without other modifiers generally means at least two recitations, or two or more recitations). In addition, in those cases where a convention similar to "at least one of A, B, and C, etc." is used, generally, such construction is intended to have the meaning that those skilled in the art will understand the convention (e.g., "a system having at least one of A, B, and C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those cases where a convention similar to "at least one of A, B, or C, etc." is used, generally, such construction is intended to have the meaning that those skilled in the art will understand the convention (e.g., "a system having at least one of A, B, or C" will include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art will also understand that, generally, unless the context indicates otherwise, transitional words and / or phrases presenting two or more alternative terms in the detailed description, claims, or drawings should be understood to include the possibility of including one of the terms, any one of the terms, or both of the terms. For example, the phrase "A or B" will generally be understood to include the possibility of "A" or "B" or "A and B".

[0395] For the attached claims, those skilled in the art will understand that the operations described therein can generally be performed in any order. In addition, although multiple operation flow charts are listed in (one or more) sequences, it should be understood that the multiple operations can be performed in other orders different from the order shown, or the multiple operations can be performed simultaneously. Unless the context otherwise provides, examples of such alternative sorting may include overlapping, interlaced, interrupted, reordered, incremental, preparatory, supplementary, simultaneous, reverse, or other changed sorting. In addition, unless the context otherwise provides, terms such as "responsive to", "related to" or other past tense adjectives are generally not intended to exclude such variants.

[0396] It is worth mentioning that any reference to "one aspect", "an aspect", "an example", or "an example" means that the specific features, structures, or characteristics described in conjunction with the aspect are included in at least one aspect. Therefore, the phrases "in one aspect", "in an aspect", "in an example", or "in an example" appearing in different places throughout the specification do not necessarily refer to the same aspect. In addition, the specific features, structures, or characteristics may be combined in any suitable manner in one or more aspects.

[0397] Any patent application, patent, non-patent publication or other public material referred to in this specification and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material is inconsistent herewith. Therefore, and to the extent necessary, the disclosure explicitly listed herein supersedes any conflicting material incorporated herein by reference. Any material or portion thereof that is allegedly incorporated herein by reference but conflicts with existing definitions, statements or other public material listed herein will be incorporated only to the extent that there is no conflict between the incorporated material and the existing public material.

[0398] In summary, many of the benefits resulting from the adoption of the concepts described herein have been described. For purposes of illustration and description, one or more of the above specific embodiments have been provided. These specific embodiments are not intended to be exhaustive or limited to the precise forms disclosed by the invention. The invention may be modified or varied in accordance with the above teachings. The one or more forms selected and described are intended to illustrate the principles and practical applications so that one of ordinary skill in the art can utilize the multiple forms and various modifications suitable for the intended specific use. The claims filed herewith are intended to define the full scope.

Claims

1. A surgical system comprising: A first device, the first device comprising a first control circuit and a display; a second device configured to perform a surgical function, wherein the second device includes a second control circuit in signal communication with the first control circuit, wherein an image of the second device is viewable on the display of the first device, and wherein the second control circuit is configured to selectively switch the second device back and forth between an auxiliary mode of operation and a primary mode of operation, wherein the second device acts as a first input device such that the image of the second device on the display is configured to interact with the image on the display to highlight unique anatomical features in the display, wherein the second device is configured to control the surgical function, and wherein the ability of the second device to perform the surgical function is not exercised in the auxiliary mode of operation; as well as A second input device, the second input device is constructed to allow clinical personnel to control the first device, the second input device includes two wearable devices, wherein the two wearable devices can be worn on the left hand and the right hand of the clinical personnel, respectively, and the two wearable devices are configured to be able to determine on which hand of the clinical personnel the second device is located.

2. The surgical system of claim 1, wherein: Wherein the second device comprises an end effector positioned within the sterile field, and wherein an image of the end effector is visible on the display.

3. The surgical system of claim 2, wherein: The secondary operating mode comprises a cursor mode, and wherein the primary operating mode comprises a tissue treatment mode.

4. A surgical system comprising: a first device, the first device comprising a first control circuit; as well as a second device configured to perform a surgical function, wherein the second device includes a second control circuit in signal communication with the first control circuit, and wherein the second control circuit is configured to selectively switch the second device back and forth between an auxiliary operating mode and a primary operating mode, wherein the second device acts as a first input device such that an image of the second device on a display of the first device is configured to interact with an image on the display such that unique anatomical features are highlighted in the display, and wherein the second device is configured to control the surgical function in the primary operating mode, wherein the ability of the second device to perform the surgical function is not exercised in the auxiliary operating mode, wherein the second device includes a handle including an input switch movable between a first position and a second position, and wherein the first position corresponds to the primary operating mode and the second position corresponds to the auxiliary operating mode; and A second input device, the second input device is constructed to allow clinical personnel to control the first device, the second input device includes two wearable devices, wherein the two wearable devices can be worn on the left hand and the right hand of the clinical personnel, respectively, and the two wearable devices are configured to be able to determine on which hand of the clinical personnel the second device is located.

5. The surgical system of claim 4, wherein: The second control circuit is configured to switch back and forth between the primary mode of operation and the secondary mode of operation in response to a clinician's voice command.

6. A surgical system comprising: a first device, the first device comprising a first control circuit and a display; a second device, the second device comprising an end effector, the end effector being configured to be capable of performing a surgical function in a sterile field, wherein the second device comprises a second control circuit in signal communication with the first control circuit, and wherein the second control circuit is configured to be capable of selectively switching the second device back and forth between an auxiliary operating mode and a primary operating mode, wherein in the auxiliary operating mode, the second device acts as a first input device such that an image of the second device on the display is configured to be capable of interacting with an image on the display such that unique anatomical features are highlighted in the display, wherein in the primary operating mode, the second device is configured to be capable of controlling the surgical function, wherein the ability of the second device to perform the surgical function is not exercised in the auxiliary operating mode, and wherein movement of the end effector within the sterile field is configured to be capable of dragging and dropping icons on the display in the auxiliary operating mode; and A second input device, the second input device is constructed to allow clinical personnel to control the first device, the second input device includes two wearable devices, wherein the two wearable devices can be worn on the left hand and the right hand of the clinical personnel, respectively, and the two wearable devices are configured to be able to determine on which hand of the clinical personnel the second device is located.

7. A surgical system comprising: a first device, the first device comprising a first control circuit and a display; a second device, the second device comprising an end effector, the end effector being configured to be capable of performing a surgical function in the sterile field, wherein the second device comprises a second control circuit in signal communication with the first control circuit, and wherein the second control circuit is configured to be capable of selectively switching the second device back and forth between an auxiliary mode of operation and a primary mode of operation, wherein the second device acts as a first input device such that an image of the second device on the display is configured to be capable of interacting with an image on the display such that a unique anatomical feature is highlighted in the display, and wherein the second device is configured to be capable of controlling the surgical function in the primary mode of operation, wherein the ability of the second device to perform the surgical function is not exercised in the auxiliary mode of operation, and wherein the image of the end effector on the display is configured to be capable of selecting an anatomical feature from the sterile field on the display in the auxiliary mode of operation; as well as A second input device, the second input device is constructed to allow clinical personnel to control the first device, the second input device includes a wearable device, wherein the wearable devices are two and can be worn on the left hand and the right hand of the clinical personnel respectively, and the two wearable devices are configured to be able to determine on which hand of the clinical personnel the second device is located.

8. A surgical system comprising: a first device, the first device comprising a first control circuit and a display; a second device, the second device comprising an end effector configured to perform a surgical function in a sterile field, wherein the second device comprises a second control circuit in signal communication with the first control circuit, and wherein the second control circuit is configured to selectively switch the second device back and forth between an auxiliary operating mode and a primary operating mode, wherein in the auxiliary operating mode, the second device acts as an input device such that an image of the second device on the display is configured to interact with an image on the display such that unique anatomical features are highlighted in the display, wherein in the primary operating mode, the second device is configured to control the surgical function, wherein the ability of the second device to perform the surgical function is not exercised in the auxiliary operating mode, and wherein the second device comprises an ultrasonic instrument configured to apply ultrasonic vibrations to tissue, wherein the ultrasonic instrument comprises a first actuation button and a second actuation button, wherein in the primary operating mode, the first actuation button is configured to actuate a first energy level and the second actuation button is configured to actuate a second energy level, and wherein in the auxiliary operating mode, the first actuation button comprises a first cursor button and the second actuation button comprises a second cursor button; and A second input device, the second input device is constructed to allow clinical personnel to control the first device, the second input device includes two wearable devices, wherein the two wearable devices can be worn on the left hand and the right hand of the clinical personnel, respectively, and the two wearable devices are configured to be able to determine on which hand of the clinical personnel the second device is located.

9. A surgical system comprising: An imaging system, the imaging system comprising a camera and a display screen; as well as A surgical device configured to perform a surgical function, wherein the surgical device comprises a control circuit comprising a processor and a memory communicatively coupled to the processor, the memory storing instructions executable by the processor to: receiving an input signal; switching between a first operating mode and a second operating mode in response to the input signal; actuating the surgical function in the first operating mode; as well as Controlling the display screen in the second operating mode, wherein the ability of the surgical device to perform the surgical function is not exercised in the second operating mode, wherein in the second operating mode, the surgical device acts as an input device so that the image of the surgical device on the display screen is configured to be able to interact with the image on the display screen, thereby highlighting unique anatomical features in the display screen, wherein the surgical system also includes a second input device, the second input device is constructed to allow clinical personnel to control the imaging system, the second input device includes two wearable devices, wherein the two wearable devices can be worn on the left and right hands of the clinical personnel, respectively, and the two wearable devices are configured to be able to determine on which hand of the clinical personnel the surgical device is positioned.

10. The surgical system of claim 9, wherein: The surgical device is configured to control the display screen through the surgical barrier.

11. The surgical system of claim 9, wherein: The display screen comprises a video monitor in an operating room, and wherein the surgical device comprises a laparoscopic device including an end effector positioned within a patient in the operating room.

12. The surgical system of claim 11, wherein: The camera is configured to be able to track the end effector within the patient's body.

13. The surgical system of claim 12, wherein: In the second operating mode, the end effector is configured to function as a cursor to interact with one or more icons on the video monitor.

14. The surgical system of claim 12, wherein: In the second mode of operation, the end effector is configured to function as a cursor to interact with video feedback on the video monitor.

15. The surgical system of claim 9, wherein: The surgical device includes a handle including an input switch movable between a first position and a second position, and wherein the first position corresponds to the first mode of operation and the second position corresponds to the second mode of operation.

16. The surgical system of claim 9, wherein: The control circuit is configured to switch back and forth between the first mode of operation and the second mode of operation in response to a clinician's voice command.

Images