A surgical system for generating a three-dimensional construct of a dissected organ and coupling identified anatomical structures with the three-dimensional construct

Abstract

A surgical system for use in a surgical procedure is disclosed. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical organ targeted by the surgical procedure, generate a virtual three-dimensional (3D) construct of at least a portion of the anatomical organ based on visualization data from the at least one imaging device, identify anatomical structures relevant to the surgical procedure from the visualization data from the at least one imaging device, couple the anatomical structures to the virtual 3D construct, and superimpose a layout plan of the surgical procedure determined based on the anatomical structures onto the virtual 3D construct.

Description

BACKGROUND

[0001] Surgical systems often incorporate imaging systems that can allow a clinician to view a surgical site and / or one or more portions thereof, for example, on one or more displays, such as monitors. The displays can be local and / or remote to the operating room. The imaging systems can include scopes with cameras that view the surgical site and transmit a view to a display that can be viewed by the clinician. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastro-duodenoscopes (gastroscope), endoscopes, laryngoscopes, nasopharyngo- nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. The imaging systems can be limited by the information they are able to identify and / or communicate to the clinician. For example, certain imaging systems can be unable to identify certain hidden structures, physical contours, and / or dimensions within a three-dimensional space intraoperatively. Additionally, certain imaging systems can be unable to transmit and / or communicate certain information to the clinician intraoperatively. SUMMARY

[0002] In one general aspect, a surgical system for a surgical procedure is disclosed. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical organ targeted by the surgical procedure, generate a virtual three-dimensional (3D) construct of at least a portion of the anatomical organ based on visualization data from the at least one imaging device, identify anatomical structures related to the surgical procedure from the visualization data from the at least one imaging device, couple the anatomical structures to the virtual 3D construct, and superimpose a layout plan of the surgical procedure determined based on the anatomical structures onto the virtual 3D construct.

[0003] In another general aspect, a surgical system for use with a surgical instrument in a surgical procedure is disclosed. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical organ targeted by the surgical procedure, identify anatomical structures related to the surgical procedure from visualization data from the at least one imaging device, and recommend a surgical resection path for removal of a portion of the anatomical organ by the surgical instrument. The surgical resection path is determined based on anatomical structures.

[0004] In yet another general aspect, a surgical system for use with a surgical instrument in a surgical procedure is disclosed. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical organ targeted by the surgical procedure, identify anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device, recommend a surgical resection path for removal of a portion of the anatomical organ by the surgical instrument, and modify the surgical resection path during the surgical procedure. The surgical resection path is determined based on the anatomical structures. BRIEF DESCRIPTION OF DRAWINGS

[0005] The novel features of the various aspects are set forth with particularity in the appended claims. These aspects, however, both as to organization and methods of operation, can be better understood by reference to the following description, taken in conjunction with the accompanying drawings as follows, wherein:

[0006] FIG. 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device configured to identify critical structures beneath a tissue surface in accordance with at least one aspect of the present disclosure.

[0007] FIG. 2 is a schematic diagram of a control system for a surgical visualization system in accordance with at least one aspect of the present disclosure.

[0008] FIG. 2A is shown a control circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure.

[0009] FIG. 2B is shown a combinational logic circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure.

[0010] FIG. 2C is shown a sequential logic circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure.

[0011] FIG. 3 is a schematic diagram of a surgical device, an imaging device, and a critical structure to triangulate to determine a depth d of the critical structure beneath a tissue surface in accordance with at least one aspect of the present disclosure. A FIG. 1

[0012] FIG. 4 is a schematic diagram of a surgical visualization system configured to identify critical structures beneath a tissue surface in accordance with at least one aspect of the present disclosure, wherein the surgical visualization system includes a pulsed light source to determine a depth d of the critical structures beneath the tissue surface A ​​​

[0013] FIG. 5 is a schematic of a surgical visualization system including an imaging device and a surgical device configured to identify critical structures beneath a tissue surface according to at least one aspect of the present disclosure.

[0014] FIG. 6 is a schematic of a surgical visualization system including a three-dimensional camera configured to identify critical structures embedded within a tissue according to at least one aspect of the present disclosure.

[0015] FIG. 7A and FIG. 7B is a view of a critical structure taken by a three-dimensional camera of FIG. 6 , wherein FIG. 7A is a view from a left side lens of the three-dimensional camera, and FIG. 7B is a view from a right side lens of the three-dimensional camera.

[0016] FIG. 8 is a schematic of a surgical visualization system according to at least one aspect of the present disclosure FIG. 6 , wherein a camera-critical structure distance d w may be determined from the three-dimensional camera to the critical structure.

[0017] FIG. 9 is a schematic of a surgical visualization system utilizing two cameras to determine the location of a critical structure embedded therein according to at least one aspect of the present disclosure.

[0018] FIG. 10A is a schematic of a surgical visualization system utilizing a camera that is axially moved between a plurality of known locations to determine the location of a critical structure embedded therein according to at least one aspect of the present disclosure.

[0019] FIG. 10B is a schematic of a surgical visualization system according to at least one aspect of the present disclosure FIG. 10A , wherein a camera is axially and rotationally moved between a plurality of known locations to determine the location of a critical structure embedded therein.

[0020] FIG. 11 is a schematic of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0021] FIG. 12 is a schematic of a structured light source for a surgical visualization system according to at least one aspect of the present disclosure.

[0022] FIG. 13A is a plot of the absorption coefficient of various biological materials at different wavelengths according to at least one aspect of the present disclosure.

[0023] FIG. 13B is a schematic illustration of visualizing an anatomical structure by a spectral surgical visualization system according to at least one aspect of the present disclosure.

[0024] FIG. 13C-13E depicts an exemplary hyperspectral signature for distinguishing anatomical structures from obscuring matter according to at least one aspect of the present disclosure, wherein FIG. 13C is a graphical representation of a ureter signature versus obscuring matter, FIG. 13D is a graphical representation of an artery signature versus obscuring matter, and FIG. 13E is a graphical representation of a nerve signature versus obscuring matter.

[0025] FIG. 14 is a schematic illustration of a near-infrared (NIR) time-of-flight measurement system configured to sense distance from critical anatomical structures according to at least one aspect of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) positioned on a common device.

[0026] FIG. 15 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. FIG. 1 is a schematic illustration of a transmitted wave, a received wave, and a delay between the transmitted wave and the received wave of the NIR time-of-flight measurement system of

[0027] FIG. 15 shows a NIR time-of-flight measurement system configured to sense distance from different structures according to at least one aspect of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) on separate devices.

[0028] FIG. 14 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0029] FIG. 14 is a surgical system for performing a surgical procedure in an operating room according to at least one aspect of the present disclosure.

[0030] FIG. 15 shows a computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0031] FIG. 14 shows a diagram of a situational awareness surgical system according to at least one aspect of the present disclosure.

[0032] FIG. 14 shows a timeline depicting situational awareness of a hub according to at least one aspect of the present disclosure.

[0033] FIG. 16is a logic flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for associating visualization data with instrument data.

[0034] FIG. 1 is a schematic view of a surgical instrument in accordance with at least one aspect of the present disclosure.

[0035] FIG. 16 is a graph depicting a composite data set and a closure force (“FTC”) and firing force (“FTF”) virtual gauge in accordance with at least one aspect of the present disclosure.

[0036] FIG. 16 shows a normal view of a screen of a visualization system displaying a real-time feed of an end effector in a surgical field of view of a surgical procedure in accordance with at least one aspect of the present disclosure.

[0037] FIG. 16 shows an augmented view of a screen of a visualization system displaying a real-time feed of an end effector in a surgical field of view of a surgical procedure in accordance with at least one aspect of the present disclosure.

[0038] Surgical hub system is a logic flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for synchronizing motion of a virtual representation of an end effector component with actual motion of the end effector component.

[0039] FIG. 17-19 shows an anatomical structure in a body wall and a cavity beneath the body wall with a trocar penetrating the body wall into the cavity and a screen displaying a distance of the trocar from the anatomical structure, a risk associated with presenting a surgical instrument through the trocar, and an estimated operating time associated therewith in accordance with at least one aspect of the present disclosure.

[0040] FIG. 17 shows a virtual three-dimensional (“3D”) construct of a stomach exposed to structured light from a structured light projector in accordance with at least one aspect of the present disclosure.

[0041] FIG. 17 is a logic flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for associating visualization data with instrument data, wherein blocks with dashed lines represent alternative implementations of the process.

[0042] FIG. 18 shows a virtual 3D construct of a stomach exposed to structured light from a structured light projector in accordance with at least one aspect of the present disclosure.

[0043] FIG. 18is a logical flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for recommending a resection path for removing a portion of an anatomical organ, where blocks with dashed lines represent alternative implementations of the process.

[0044] FIG. 18 shows a real-time view of a surgical field on a screen of a visualization system at the beginning of a surgical procedure in accordance with at least one aspect of the present disclosure.

[0045] FIG. 18 is a logical flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for recommending a resection path for removing a portion of an anatomical organ. FIG. 19 is a magnified view of a portion of the surgical field of

[0046] FIG. 19 shows a real-time view of a surgical field of Situational awareness at forty-three minutes after the beginning of a surgical procedure in accordance with at least one aspect of the present disclosure.

[0047] FIG. 17-19 is a magnified view of a portion of the surgical field of FIG. 20 in accordance with at least one aspect of the present disclosure outlining a modification to the recommended surgical resection path.

[0048] FIG. 2 is a logical flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for presenting parameters of a surgical instrument onto or near a recommended surgical resection path, where blocks with dashed lines represent alternative implementations of the process.

[0049] FIG. 21 shows a virtual 3D construct of a stomach of a patient undergoing a sleeve gastrectomy in accordance with at least one aspect of the present disclosure.

[0050] FIG. 1-11 shows a complete virtual resection of the stomach of FIG. 2 in accordance with at least one aspect of the present disclosure.

[0051] FIG. 17 shows a firing of a surgical stapling instrument in accordance with at least one aspect of the present disclosure.

[0052] FIG. 22 is a logical flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for adjusting a firing speed of a surgical instrument.

[0053] FIG. 1-18 is a logical flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for recommended staple cartridge placement along a recommended surgical resection path.

[0054] FIG. 1 is a logic flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for recommending surgical resection of an organ portion.

[0055] FIG. 1 is a logic flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for estimating a reduction in organ volume due to removal of a selected portion of an organ.

[0056] FIG. 1 shows a patient lung exposed to structured light, including a portion to be resected during a surgical procedure, in accordance with at least one aspect of the present disclosure.

[0057] FIG. 3-4 shows the patient lung of FIG. 13A-16 after resection of the portion, in accordance with at least one aspect of the present disclosure.

[0058] FIG. 12 shows a plot of lung volume peaks measured from the patient lung of FIG. 23 and FIG. 23 before and after resection of the portion of the lung, in accordance with at least one aspect of the present disclosure.

[0059] FIG. 2A shows a plot of partial pressure of carbon dioxide (“PCO2”) measured in the patient lung before, immediately after, and one minute after resection of a portion of the lung, in accordance with at least one aspect of the present disclosure.

[0060] FIG. 2B is a logic flow diagram of a process in accordance with at least one aspect of the present disclosure depicting a control program or logic configuration for detecting tissue abnormalities using visualized data and non-visualized data.

[0061] FIG. 2C shows a right lung in a first state, in accordance with at least one aspect of the present disclosure, in which an imaging device emits a pattern of light onto a surface thereof.

[0062] FIG. 19 shows the right lung of FIG. 24 in a second state, in accordance with at least one aspect of the present disclosure, in which an imaging device emits a pattern of light onto a surface thereof.

[0063] FIG. 24 shows a top portion of the right lung of FIG. 24 in accordance with at least one aspect of the present disclosure.

[0064] FIG. 24 shows a plot of a lung volume peak measured from the patient lung ofFIG. 24 a top portion of the right lung. DETAILED DESCRIPTION

[0065] Applicant of the present application owns the following U.S. Patent Applications that are each herein incorporated by reference in their respective entireties:

[0066] • Attorney Docket No. END9228USNP1 / 190580-1M, entitled “METHOD OF USING

[0067] • Attorney Docket No. END9227USNP1 / 190579-1, entitled “ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM”;

[0068] • Attorney Docket No. END9226USNP1 / 190578-1, entitled “SURGICAL SYSTEM CONTROL BASED ON MULTIPLE SENSED PARAMETERS”;

[0069] • Attorney Docket No. END9225USNP1 / 190577-1, entitled “ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS”;

[0070] • Attorney Docket No. END9224USNP1 / 190576-1, entitled “ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS”;

[0071] • Attorney Docket No. END9223USNP1 / 190575-1, entitled “SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA”;

[0072] • Attorney Docket No. END9221USNP1 / 190573-1, entitled SURGICAL SYSTEM FOR OVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN;

[0073] • Attorney Docket No. END9220USNP1 / 190572-1, entitled SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS;

[0074] • Attorney Docket No. END9219USNP1 / 190571-1, entitled SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE;

[0075] • Attorney Docket No. END9218USNP1 / 190570-1, entitled VISUALIZATION SYSTEMS USING STRUCTURED LIGHT;

[0076] • Attorney Docket No. END9217USNP1 / 190569-1, entitled DYNAMIC SURGICAL VISUALIZATION SYSTEMS; and

[0077] • Attorney Docket No. END9216USNP1 / 190568-1, entitled ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM.

[0078] Applicant of the present application owns the following U.S. Patent Applications that were filed on March 15, 2019 and which are each herein incorporated by reference in their respective entirety:

[0079] • U.S. Patent Application Serial No. 16 / 354,417, entitled INPUT CONTROLS FOR ROBOTIC SURGERY;

[0080] • U.S. Patent Application Serial No. 16 / 354,420, titled DUAL MODE CONTROLS FOR ROBOTIC SURGERY;

[0081] • U.S. Patent Application Serial No. 16 / 354,422, titled MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY;

[0082] • U.S. Patent Application Serial No. 16 / 354,440, titled ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY;

[0083] • U.S. Patent Application Serial No. 16 / 354,444, titled ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE;

[0084] • U.S. Patent Application Serial No. 16 / 354,454, titled ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS;

[0085] • U.S. Patent Application Serial No. 16 / 354,461, titled SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS;

[0086] • U.S. Patent Application Serial No. 16 / 354,470, titled SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS;

[0087] • U.S. Patent Application Serial No. 16 / 354,474, titled ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES;

[0088] • U.S. Patent Application Serial No. 16 / 354,478, titled ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK; and

[0089] • U.S. Patent Application Serial No. 16 / 354,481, titled JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS.

[0090] Applicant of the present application owns the following U.S. Patent Applications, filed on September 11, 2018, each of which is herein incorporated by reference in its entirety:

[0091] • U.S. Patent Application Serial No. 16 / 128,179, titled SURGICAL VISUALIZATION PLATFORM;

[0092] • U.S. Patent Application Serial No. 16 / 128,180, titled CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE;

[0093] • U.S. Patent Application Serial No. 16 / 128,198, titled SINGULAR EMR SOURCE EMITTER ASSEMBLY;

[0094] • U.S. Patent Application Serial No. 16 / 128,207, titled COMBINATION EMITTER AND CAMERA ASSEMBLY;

[0095] • U.S. Patent Application Serial No. 16 / 128,176, titled SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES;

[0096] • U.S. Patent Application Serial No. 16 / 128,187, titled SURGICAL VISUALIZATION OF MULTIPLE TARGETS;

[0097] • U.S. Patent Application Serial No. 16 / 128,192, titled VISUALIZATION OF SURGICAL DEVICES;

[0098] • U.S. Patent Application Serial No. 16 / 128,163, titled OPERATIVE COMMUNICATION OF LIGHT;

[0099] • U.S. Patent Application Serial No. 16 / 128,197, titled ROBOTIC LIGHT

[0100] • U.S. Patent Application Serial No. 16 / 128,164, titled SURGICAL VISUALIZATION

[0101] • U.S. Patent Application Serial No. 16 / 128,193, titled SURGICAL VISUALIZATION AND

[0102] • U.S. Patent Application Serial No. 16 / 128,195, titled INTEGRATION OF IMAGING DATA

[0103] • U.S. Patent Application Serial No. 16 / 128,170, titled ROBOTICALLY-ASSISTED

[0104] • U.S. Patent Application Serial No. 16 / 128,183, titled SAFETY LOGIC FOR SURGICAL

[0105] • U.S. Patent Application Serial No. 16 / 128,172, titled ROBOTIC SYSTEM WITH SEPARATE

[0106] • U.S. Patent Application Serial No. 16 / 128,185, titled FORCE SENSOR THROUGH

[0107] The Applicant of the present application also owns the following U.S. Patent Applications, each of which is herein incorporated by reference in its entirety:

[0108] • U.S. Patent Application Serial No. 15 / 940,627, titled DRIVE ARRANGEMENTS FOR

[0109] • U.S. Patent Application Serial No. 15 / 940,711, titled SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS, now U.S. Patent Application Publication No. 2019 / 0201120;

[0110] • U.S. Patent Application Serial No. 15 / 940,711, titled SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS, now U.S. Patent Application Publication No. 2019 / 0201120; and

[0111] • U.S. Patent Application Serial No. 15 / 940,722, titled CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY, now U.S. Patent Application Publication No. 2019 / 0200905.

[0112] Applicant of the present application owns the following U.S. Patent Applications, filed on December 4, 2018, the disclosure of each of which is herein incorporated by reference in its entirety:

[0113] • U.S. Patent Application Serial No. 16 / 209,395, titled METHOD OF HUB

[0114] • U.S. Patent Application Serial No. 16 / 209,403, titled METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB, now U.S. Patent Application Publication No. 2019 / 0206569;

[0115] • U.S. Patent Application Serial No. 16 / 209,407, titled METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL, now U.S. Patent Application Publication No. 2019 / 0201137;

[0116] • U.S. Patent Application Serial No. 16 / 209,416, titled METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS, now U.S. Patent Application Publication No. 2019 / 0206562;

[0117] • U.S. Patent Application Serial No. 16 / 209,423, titled METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS, now U.S. Patent Application Publication No. 2019 / 0200981;

[0118] • U.S. Patent Application Serial No. 16 / 209,427, titled METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES, now U.S. Patent Application Publication No. 2019 / 0208641;

[0119] • U.S. Patent Application Serial No. 16 / 209,433, titled METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB, now U.S. Patent Application Publication No. 2019 / 0201594;

[0120] • U.S. Patent Application Serial No. 16 / 209,447, titled METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB, now U.S. Patent Application Publication No. 2019 / 0201045;

[0121] • U.S. Patent Application Serial No. 16 / 209,453, titled METHOD FOR CONTROLLING SMART ENERGY DEVICES, now U.S. Patent Application Publication No. 2019 / 0201046;

[0122] • U.S. Patent Application Serial No. 16 / 209,458, titled METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE, now U.S. Patent Application Publication No. 2019 / 0201047;

[0123] • U.S. Patent Application Serial No. 16 / 209,465, titled METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION, now U.S. Patent Application Publication No. 2019 / 0206563;

[0124] • U.S. Patent Application Serial No. 16 / 209,478, titled METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE, now U.S. Patent Application Publication No. 2019 / 0104919;

[0125] • U.S. Patent Application Serial No. 16 / 209,490, titled METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION, now U.S. Patent Application Publication No. 2019 / 0206564; and

[0126] • U.S. Patent Application Serial No. 16 / 209,491, titled METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS, now U.S. Patent Application Publication No. 2019 / 0200998.

[0127] Before the various aspects of the surgical visualization platform are described in detail, it should be noted that the illustrative examples are not limited in application or use to the details of construction and arrangement of parts illustrated in the drawings and description. The illustrative examples can be implemented or incorporated in other aspects, variations and modifications, and can be practiced or carried out in various ways. Further, unless otherwise indicated, the terms and expressions employed herein have their article- selected meanings as commonly understood by one of ordinary skill in the art in the field of the illustrative examples and are not to be construed as strictly limited. Also, it is to be understood that one or more of the following-described aspects, expressions of aspects, and / or examples can be combined with any one or more of the other following-described aspects, expressions of aspects, and / or examples.

[0128] FIG. 24

[0129] The present disclosure relates to a surgical visualization platform that utilizes “digital surgery” to obtain additional information about a patient’s anatomy and / or surgery. The surgical visualization platform is further configured to communicate data and / or information to one or more clinicians in a helpful manner. For example, various aspects of the present disclosure provide improved visualization of a patient’s anatomy and / or surgery.

[0130] “Digital surgery” can encompass robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and more. Although various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, the surgical visualization platforms are not limited to use with robotic surgical systems. In certain instances, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0131] In certain instances, a surgical system incorporating a surgical visualization platform can implement intelligent dissection in order to identify and avoid critical structures. Critical structures include anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors, among other anatomical structures. In other instances, a critical structure can be a foreign structure in the field of dissection, such as a surgical device, a surgical fastener, a clip, a tack, a probe, a band, and / or a plate, for example. Critical structures can be determined based on different patients and / or different procedures. Exemplary critical structures are also described herein. For example, intelligent dissection techniques can provide improved intraoperative guidance for dissection, and / or can implement intelligent decision making with critical anatomical structure detection and avoidance techniques.

[0132] Surgical systems incorporating surgical visualization platforms can also implement intelligent stapling techniques that utilize improved workflows to provide more consistent stapling at optimal locations. Cancer localization techniques can also be improved utilizing various surgical visualization platforms and procedures described herein. For example, cancer localization techniques can identify and track cancer locations, orientations, and boundaries thereof. In certain instances, cancer localization techniques can compensate for movement of tools, patients, and / or patient anatomy during a surgical procedure in order to provide guidance to a clinician back to a point of interest.

[0133] In certain aspects of the present disclosure, surgical visualization platforms can provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue types and health without the need for physical palpation, particularly when dissecting and / or placing suturing devices within tissue. Certain tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, surgical visualization platforms can preoperatively localize, map, and ideally diagnose lymphatic systems and / or lymph nodes involved, for example, in cancer diagnosis and staging.

[0134] During a surgical procedure, information available to a clinician via the naked eye and / or imaging systems can provide an incomplete view of a surgical site. For example, certain structures, such as structures embedded or buried within an organ, can be at least partially obscured or hidden from view. Additionally, certain dimensions and / or relative distances can be difficult to perceive with the naked eye and / or to ascertain with existing sensor systems. Furthermore, certain structures can move preoperatively (e.g., before a surgical procedure but after a preoperative scan) and / or intraoperatively. In such instances, a clinician can not be able to accurately determine the location of a critical structure intraoperatively.

[0135] When the location of a critical structure is uncertain and / or when the proximity between a critical structure and a surgical tool is unknown, a clinician’s decision-making process can be hindered. For example, a clinician can avoid certain areas in order to avoid inadvertently dissecting a critical structure; however, the areas avoided can be unnecessarily large and / or at least partially misplaced. Due to uncertainty and / or over / undue caution, a clinician can not be able to access certain desired areas. For example, over-caution can cause a clinician to leave a portion of a tumor and / or other undesirable tissue in an attempt to avoid a critical structure, even if the critical structure is not in that particular area and / or would not be negatively impacted by the clinician working in that particular area. In certain instances, surgical outcomes can be improved by increasing knowledge and / or certainty, which can make a surgeon more accurate with respect to particular anatomical areas and, in certain instances, make a surgeon less conservative / more aggressive.

[0136] In various aspects, the present disclosure provides surgical visualization systems for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what a clinician sees with the “naked eye” and / or beyond what an imaging system can identify and / or communicate to the clinician. The various surgical visualization systems can enhance and augment what a clinician can know prior to treatment of tissue (e.g., dissection) and, thus, can improve outcomes in various situations.

[0137] For example, a visualization system can include a first light emitter configured to emit a plurality of spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, a molecular response to the spectral waves (spectral imaging), and / or the light pattern. It should be noted that throughout the disclosure below, unless specifically mentioned visible light, any reference to “light” can include photons in the visible and / or non-visible portions of the electromagnetic radiation (EMR) or EMR wavelength spectrum. The surgical visualization system can also include an imaging system and a control circuit in signal communication with the receivers and the imaging system. Based on the output from the receivers, the control circuit can determine a geometric surface map (i.e., three-dimensional surface topography) of a visible surface at a surgical site and one or more distances relative to the surgical site. In certain situations, the control circuit can determine one or more distances to at least partially hidden structures. Further, the imaging system can communicate the geometric surface map and the one or more distances to a clinician. In such situations, the enhanced view of the surgical site provided to the clinician can provide a representation of hidden structures within the relevant environment of the surgical site. For example, the imaging system can virtually enhance the hidden structures on the geometric surface map of hidden and / or obstructing tissue similar to a line drawn on the ground to indicate a utility line below the surface. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to the visible obstructing tissue and / or to the at least partially hidden structures and / or the depth of the hidden structures below the visible surface of the obstructing tissue. For example, the visualization system can determine a distance relative to an enhanced line on the surface of the visible tissue and communicate the distance to the imaging system.

[0138] In various aspects of the present disclosure, surgical visualization systems for intraoperative identification and avoidance of critical structures are disclosed. Such surgical visualization systems can provide valuable information to a clinician during a surgical procedure. Thus, for example, a clinician knowing that a surgical visualization system is tracking a critical structure, such as a ureter, a particular nerve, and / or a critical blood vessel, for example, that can be approached during dissection, can confidently maintain momentum throughout a surgical procedure. In one aspect, the surgical visualization system can provide an indication to the clinician for the clinician to pause and / or slow the surgical procedure and assess proximity to the critical structure to prevent accidental damage thereto for a sufficient amount of time. The surgical visualization system can provide an ideal, optimized, and / or customizable amount of information to the clinician to allow the clinician to confidently and / or quickly move through tissue while avoiding accidental damage to healthy tissue and / or critical structures and, thus, minimizing the risk of injury caused by the surgical procedure.

[0139] FIG. 22 is a schematic illustration of a surgical visualization system 100 in accordance with at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of a critical structure 101 within a dissection field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In certain instances, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near real-time information regarding proximity data, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured for intraoperative identification of critical structures and / or to facilitate avoidance of critical structures 101 by surgical devices. For example, by identifying critical structures 101, a clinician can avoid manipulating surgical devices around critical structures 101 and / or areas within a predetermined proximity of critical structures 101 during a surgical procedure. For example, a clinician can avoid dissecting and / or avoid dissecting near veins, arteries, nerves, and / or blood vessels, for example, that are identified as critical structures 101. In various instances, the critical structures 101 can be determined based on different patients and / or different procedures.

[0140] The surgical visualization system 100 incorporates distance sensor system 104 in combination with tissue identification and geometric surface mapping. In combination, these features of the surgical visualization system 100 can determine the location of critical structures 101 within the anatomical field and / or the proximity of the surgical device 102 to the surface 105 of visible tissue and / or to the critical structures 101. In addition, the surgical visualization system 100 includes an imaging system that includes, for example, an imaging device 120, such as a camera, configured to provide real-time views of the surgical site. In various instances, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. The views from the imaging device 120 can be provided to the clinician, and in various aspects of the present disclosure, these views can be augmented with additional information based on tissue identification, topographic mapping, and distance sensor system 104. In such instances, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems can cooperate to provide advanced data synthesis and integrated information to the clinician intraoperatively.

[0141] The imaging device can include a camera or imaging sensor configured to detect, for example, visible light, spectral light waves (visible or non-visible), and structured light patterns (visible or non-visible). In various aspects of the present disclosure, the imaging system can include, for example, an imaging device such as an endoscope. Additionally or alternatively, the imaging system can include, for example, an imaging device such as an arthroscope, a

[0142] In various aspects of the present disclosure, the tissue identification subsystem can be implemented with a spectral imaging system. The spectral imaging system can rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled “SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING,” published March 1, 2016, which is incorporated by reference herein in its entirety.

[0143] In various aspects of the present disclosure, the surface mapping subsystem can be implemented with a light pattern system, as further described herein. Light patterns (or structured light) are used for surface mapping are known. Known surface mapping techniques can be used in the surgical visualization systems described herein.

[0144] Structured light is a process of projecting a known pattern (typically a grid or horizontal bars) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published September 7, 2017, are incorporated by reference herein in their entireties.

[0145] In various aspects of the present disclosure, a distance determination system can be incorporated into the surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of a visible surface and determine various distances relative to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to identified tissue (or other structures) at a surgical site.

[0146] FIG. 25A is a schematic view of a control system 133 that can be used with the surgical visualization system 100. The control system 133 includes a control circuit 132 in signal communication with a memory 134. The memory 134 stores instructions executable by the control circuit 132 to determine and / or identify critical structures (e.g., critical structures 101 in FIG. 25B The memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of the logic 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (e.g., a stereo camera pair) and / or a light source 146. The control circuit 132 is in signal communication with the imaging system 142 and the light source 146. FIG. 25AThe camera 144 can include one or more image sensors 135 to receive signals from various light sources (e.g., visible light, spectral imagers, three-dimensional lenses, etc.) that emit light at various visible and non-visible spectrums. The display 146 can include one or more screens or monitors to depict real, virtual, and / or virtually augmented images and / or information to one or more clinicians.

[0147] In various aspects, the heart of the camera 144 is the image sensor 135. Generally, modern image sensors 135 are solid-state electronic devices containing up to millions of discrete photodetector sites, known as pixels. Image sensor 135 technology falls into one of two categories: charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS) imagers, and more recently, short-wave infrared (SWIR) is an emerging imaging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name “sCMOS”) and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 are sensitive to wavelengths of about 350 nm to 1050 nm, but this range is often given as 400 nm to 1000 nm. Generally, CMOS sensors are more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and thus cannot distinguish colors. Thus, there are two types of color CCD cameras: single-chip and three-chip. Single-chip color CCD cameras provide a common low-cost imaging solution and use a mosaic (e.g., Bayer) optical filter to split incident light into a series of colors and employ an interpolation algorithm to resolve a full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras provide higher resolution by employing a prism to direct each portion of the incident light spectrum to a different chip. More accurate color reproduction is possible because each point in space of an object has separate RGB intensity values, rather than using algorithms to determine color. Three-chip cameras provide extremely high resolution.

[0148] The control system 133 also includes a spectral light source 150 and a structured light source 152. In some cases, a single source can be pulsed to emit wavelengths of light in the range of the spectral light source 150 and wavelengths of light in the range of the structured light source 152. Alternatively, a single light source can be pulsed to provide wavelengths of light in the invisible spectrum (e.g., infrared spectrum light) and wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various cases, the tissue identification logic 140 can identify key structures via data from the spectral light source 150 received by the image sensor 135 portion of the camera 144. The surface mapping logic 136 can determine a surface profile of the visible tissue based on reflected structured light. With time-of-flight measurements, the distance determination logic 141 can determine one or more distances to the visible tissue and / or key structures 101. One or more outputs from the surface mapping logic 136, the tissue identification logic 140, and the distance determination logic 141 can be provided to the imaging logic 138 and can be combined, blended, and / or overlaid to be conveyed to the clinician via the display 146 of the imaging system 142.

[0149] The specification now turns briefly to FIG. 26 to describe various aspects of the control circuit 132 for controlling various aspects of the surgical visualization system 100. Turning to FIG. 2A , control circuit 400 configured to control aspects of the surgical visualization system 100 according to at least one aspect of the present disclosure is shown. The control circuit 400 can be configured to implement the various processes described herein. The control circuit 400 can include a microcontroller including one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuit 404 stores machine executable instructions that, when executed by the processor 402, cause the processor 402 to execute the machine instructions to implement the various processes described herein. The processor 402 can be any of a variety of single- or multi-core processors known in the art. The memory circuit 404 can include volatile storage media and non-volatile storage media. The processor 402 can include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit can be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0150] FIG. 2BA combinational logic circuit 410 configured to control aspects of the surgical visualization system 100 is shown in accordance with at least one aspect of the present disclosure. The combinational logic circuit 410 can be configured to implement various processes described herein. The combinational logic circuit 410 can include a finite state machine that includes combinational logic 412 configured to receive data associated with a surgical instrument or tool at an input 414, process the data through the combinational logic 412, and provide an output 416.

[0151] FIG. 2C A sequential logic circuit 420 configured to control aspects of the surgical visualization system 100 is shown in accordance with at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 can include a finite state machine. The sequential logic circuit 420 can include, for example, a combinational logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store a current state of the finite state machine. In some instances, the sequential logic circuit 420 can be synchronous or asynchronous. The combinational logic 422 is configured to receive data associated with a surgical device or system from an input 426, process the data through the combinational logic 422, and provide an output 428. In other aspects, the circuit can include a combination of a processor (e.g., the processor 402 in FIG. 22 , and a finite state machine to implement various processes herein. In other aspects, the finite state machine can include a combination of a combinational logic circuit (e.g., the combinational logic circuit 410, FIG. 25B , and the sequential logic circuit 420.

[0152] Referring again to the surgical visualization system 100 in FIG. 25B , the critical structure 101 can be an anatomical structure of interest. For example, the critical structure 101 can be an anatomical structure such as a ureter, an artery such as the superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor. In other instances, the critical structure 101 can be a foreign structure in the field of dissection such as a surgical device, a surgical fastener, a clip, a tack, a probe, a band, and / or a plate, for example. Exemplary critical structures are further described herein and in the aforementioned concurrently filed U.S. Patent Applications, including, for example, U.S. Patent Application No. 16 / 128,192, entitled “VISUALIZATION OF SURGICAL DEVICES,” filed September 11, 2018, which are incorporated by reference herein in their entireties.

[0153] In one aspect, the critical structure 101 can be embedded in the tissue 103. In other words, the critical structure 101 can be positioned below the surface 105 of the tissue 103. In such cases, the tissue 103 conceals the critical structure 101 from view by the clinician. From the perspective of the imaging device 120, the critical structure 101 is also obscured by the tissue 103. The tissue 103 can be, for example, fat, connective tissue, adhesions, and / or an organ. In other cases, the critical structure 101 can be partially obscured from view.

[0154] FIG. 25B A surgical device 102 is also depicted. The surgical device 102 includes an end effector having opposing jaws extending from a distal end of a shaft of the surgical device 102. The surgical device 102 can be any suitable surgical device such as, for example, a dissector, a stapler, a grasper, a clip applier, and / or an energy device (including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector). Additionally or alternatively, the surgical device 102 can include another imaging or diagnostic modality such as, for example, an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 can be configured to enable identification of one or more critical structures 101 and proximity of the surgical device 102 to the critical structure 101.

[0155] The imaging device 120 of the surgical visualization system 100 is configured to detect various wavelengths of light such as, for example, visible light, spectral light waves (visible or non-visible), and structured light patterns (visible or non-visible). The imaging device 120 can include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 can be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 can also include a wave form sensor 122 such as a spectral image sensor, detector, and / or three-dimensional camera lens. For example, the imaging device 120 can include a right lens and a left lens that are used together to record two two-dimensional images simultaneously and, thus, generate a three-dimensional image of the surgical site, render the three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 can be configured to receive images indicative of the topography of the visible tissue and the identification and location of the hidden critical structure, as further described herein. For example, the field of view of the imaging device 120 can overlap with the pattern of light (structured light) on the surface 105 of the tissue, as shown in FIG. 1. FIG. 25B

[0156] ​In one aspect, the surgical visualization system 100 can be incorporated into a robotic system 110. For example, the robotic system 110 can include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include rigid structural members 116 and joints 118, which can include servo motor controls. The first robotic arm 112 is configured to manipulate the surgical device 102, and the second robotic arm 114 is configured to manipulate the imaging device 120. A robotic control unit can be configured to issue control motions to the robotic arms 112, 114, which can affect, for example, the surgical device 102 and the imaging device 120.

[0157] The surgical visualization system 100 also includes an emitter 106 configured to emit a pattern of light, such as a stripe, grid lines, and / or dots, to enable determination of the topography or terrain of the surface 105. For example, a projected light array 130 can be used for three-dimensional scanning and registration on the surface 105. The projected light array 130 can be emitted from the emitter 106 located, for example, in one of the surgical device 102 and / or the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is used to determine the shape defined by the surface 105 of the tissue 103 and / or the motion of the surface 105 intraoperatively. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances relative to the surface 105.

[0158] In one aspect, the imaging device 120 can also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon can be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibrational sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelengths of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 can be configured to enable identification of the type of anatomical and / or physical structure, such as the critical structure 101. Identification of the critical structure 101 can be achieved by, for example, spectroscopy, optoacoustics, and / or ultrasound. In one aspect, the wavelengths of the electromagnetic radiation 124 can be variable. The waveform sensor 122 and the optical waveform emitter 123 can include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other cases, the waveform sensor 122 and the optical waveform emitter 123 can include, for example, an optoacoustic imaging system. In other cases, the optical waveform emitter 123 can be positioned on a surgical device separate from the imaging device 120.

[0159] The surgical visualization system 100 can also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one aspect, the time-of-flight distance sensor system 104 can be a time-of-flight distance sensor system including a transmitter, such as transmitter 106, and a receiver 108 that can be positioned on the surgical device 102. In other instances, the time-of-flight transmitter can be separate from the structured light transmitter. In one general aspect, the transmitter 106 portion of the time-of-flight distance sensor system 104 can include a very tiny laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 can include a matching sensor. The time-of-flight distance sensor system 104 can detect the "time-of-flight" or the time it takes for a laser emitted by the transmitter 106 to bounce back to the sensor portion of the receiver 108. The use of a very narrow light source in the transmitter 106 enables the distance sensor system 104 to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104. Still referring to FIG. 1, the distance sensor system 104 can be used to determine the distance to the surface 105 of the tissue 103, which can be represented as d FIG. 27 , d e is the transmitter-tissue distance from the transmitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue. The distance sensor system 104 can be used to determine the transmitter-tissue distance d e . The device-tissue distance d t may be obtained from the known orientation of the transmitter 106 on the shaft of the surgical device 102 relative to the distal end of the surgical device 102. In other words, when the distance between the transmitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t may be determined from the transmitter-tissue distance d e . In certain instances, the shaft of the surgical device 102 can include one or more articulating joints and can be able to articulate relative to the transmitter 106 and the jaws. The articulating configuration can include, for example, a multi-joint vertebral structure. In certain instances, a three-dimensional camera can be used to triangulate one or more distances to the surface 105.

[0160] In various instances, the receiver 108 of the time-of-flight distance sensor system 104 can be mounted on a separate surgical device rather than the surgical device 102. For example, the receiver 108 can be mounted on a cannula or trocar through which the surgical device 102 extends to reach the surgical site. In other instances, the receiver 108 of the time-of-flight distance sensor system 104 can be mounted on a separate robotically controlled arm (e.g., the robotic arm 114), mounted on a movable arm operated by another robot, and / or mounted to an operating room (OR) table or fixture. In certain instances, the imaging device 120 includes the time-of-flight receiver 108 to determine the distance d e from the emitter 106 to the surface 105 of the tissue 103 using the line between the emitter 106 on the surgical device 102 and the imaging device 120. For example, the distance d w may be triangulated based on the known positions of the emitter 106 (on the surgical device 102) and the receiver 108 (on the imaging device 120). The three-dimensional position of the receiver 108 can be known and / or registered to the robotic coordinate plane intraoperatively.

[0161] In certain instances, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112 and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other instances, the surgical visualization system 100 can be used separate from a robotic system. In such instances, the distance sensor system 104 can be independent of the robotic system.

[0162] In certain instances, one or more of the robotic arms 112, 114 can be separate from a master robotic system used in a surgical procedure. At least one of the robotic arms 112, 114 can be positioned and registered to a particular coordinate system without servo motor control. For example, a closed loop control system and / or a plurality of sensors for the robotic arm 110 can control and / or register the position of the robotic arms 112, 114 relative to a particular coordinate system. Similarly, the position of the surgical device 102 and the imaging device 120 can be registered relative to a particular coordinate system.

[0163] Still referring to FIG. 27 , d w is the camera-key structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the critical structure 101, and d Ais the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the critical structure 101). In various aspects, the time-of-flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-critical structure distance d w The use of spectral imaging in conjunction with time-of-flight sensors is further described herein. Also, now referring to FIG. 28 In various aspects of the present disclosure, the depth d A of the critical structure 101 relative to the surface 105 of the tissue 103 can be determined by triangulation from the distance d w and the known orientation (and thus the known distance d x between) of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 to determine the distance d y which is the sum of the distances d e and d A .

[0164] Additionally or alternatively, the time-of-flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or range of waveforms) can be used to determine the camera-critical structure distance d w and a second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such cases, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0165] Additionally or alternatively, in certain cases, the distance d A may be determined by ultrasound, registered magnetic resonance imaging (MRI), or computed tomography (CT) scans. In other cases, the distance d A may be determined with spectral imaging, as the detected signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detected signal in a first manner or first amount, and collagen can reduce the detected signal in a different second manner or second amount.

[0166] Now referring to FIG. 1 a surgical visualization system 160 in which a surgical device 162 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect reflected waveforms. The optical waveform emitter 123 can be configured to emit waveforms for determining the distance d t and d wAs further described herein. In such cases, the distance d A may be determined as follows:

[0167] d A = d w - d t .

[0168] As disclosed herein, various information about the visible tissue, embedded critical structure, and surgical device can be determined by utilizing a combination approach that combines an image sensor configured to detect spectral wavelengths and structured light arrays in combination with one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays. Further, the image sensor can be configured to receive visible light and thus provide an image of the surgical site to the imaging system. Logic or algorithms are employed to identify the information received from the time-of-flight sensors, spectral wavelengths, structured light, and visible light and render a three-dimensional image of the surface tissue and underlying anatomy. In various cases, the imaging device 120 can include multiple image sensors.

[0169] Camera-critical structure distance d w may also be detected in one or more alternative ways. In one aspect, a fluorescent visualization technique such as fluorescent indocyanine green (ICG) can be utilized to illuminate the critical structure 201 as shown in FIG. 29 The camera 220 can include two optical waveform sensors 222, 224 that simultaneously take left and right images of the critical structure 201 FIG. 2A and FIG. 2B In such cases, the camera 220 can depict a glow of the critical structure 201 under the surface 205 of the tissue 203 and the distance d w may be determined from a known distance between the sensors 222 and 224. In certain cases, the distance can be more accurately determined by utilizing more than one camera or by moving the camera between multiple positions. In certain aspects, one camera can be controlled by a first robotic arm and a second camera can be controlled by another robotic arm. In such robotic systems, one camera can be a slave camera on a slave arm, for example. The slave arm and camera thereon can be programmed to track the other camera and maintain a particular distance and / or lens angle, for example.

[0170] In other aspects, the surgical visualization system 100 can employ two separate waveform receivers (i.e., cameras / image sensors) to determine d w Referring now to FIG. 2CIf the critical structure 301 or its contents (e.g., a blood vessel or blood vessel contents) can emit a signal 302, such as with fluoroscopy, the actual location can be triangulated from two separate cameras 320a, 320b at known locations.

[0171] In another aspect, referring now to FIG. 19 and FIG. 28 , the surgical visualization system can employ a dithering or moving camera 440 to determine the distance d w The camera 440 is robotically controlled such that the three-dimensional coordinates of the camera 440 at different orientations are known. In various cases, the camera 440 can be pivoted at the cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., a blood vessel or blood vessel contents) can emit a signal, such as with fluoroscopy, the actual location can be triangulated from the camera 440 moving rapidly between two or more known locations. In FIG. 14 , the camera 440 is moved axially along the axis A. More specifically, the camera 440 is translated along the axis A a distance dl closer to the critical structure 401 to a position indicated as position 440', such as by moving in and out on a robotic arm. As the camera 440 is moved the distance dl and the size of the view changes relative to the critical structure 401, the distance to the critical structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance dl) can correspond to an angle θι of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, the camera 440 can be rotated or swept along an arc between different orientations. Referring now to FIG. 28 , the camera 440 is moved axially along the axis A and rotated about the axis A by an angle θ3. The pivot point 442 for the rotation of the camera 440 is positioned at the cannula / patient interface. In Organ , the camera 440 is translated and rotated to a position 440". As the camera 440 is moved and the view edges change relative to the critical structure 401, the distance to the critical structure 401 can be calculated. In Tier 1 , the distance d2 can be, for example, 9.01 mm and the angle θ3 can be, for example, 0.9 degrees.

[0172] Tier 2A surgical visualization system 500 is depicted, which is similar to surgical visualization system 100 in many respects. In various cases, surgical visualization system 500 can be another example of surgical visualization system 100. Similar to surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light of multiple wavelengths to obtain spectral images of, for example, hidden structures. In various cases, imaging device 520 may also include a three-dimensional camera and associated electronic processing circuitry. Surgical visualization system 500 is shown intraoperatively for identifying and facilitating the avoidance of certain critical structures not visible on the surface, such as ureters 501a and blood vessels 501b in organ 503 (uterus in this example).

[0173] The surgical visualization system 500 is configured to determine the emitter-tissue distance d from the emitter 506 on the surgical device 502 to the surface 505 of the uterus 503 via structured light. e The surgical visualization system 500 is configured to be able to visualize based on the transmitter-tissue distance d. e The device extends from the surgical device 502 to the surface 505 of the uterus 503 via a tissue distance d. t The surgical visualization system 500 is also configured to determine the tissue-ureteral distance d from the ureter 501a to the surface 505. A And the camera-ureter distance d from imaging device 520 to ureter 501a w As this article discusses... Tier 3 For example, the surgical visualization system 500 may utilize, for example, spectral imaging and time-of-flight sensors to determine the distance d. w In various situations, the surgical visualization system 500 can determine (e.g., triangulation) the tissue-ureter distance d based on other distance and / or surface mapping logic components described herein. A (or depth).

[0174] See now Lung The diagram depicts a control system 600 for, for example, a surgical visualization system (such as surgical visualization system 100). For instance, the control system 600 is a conversion system that integrates spectral feature tissue recognition and structured optical tissue localization to identify key structures, particularly when these structures are obscured by other tissues such as fat, connective tissue, blood, and / or other organs. Such techniques can also be used to detect tissue variability, such as distinguishing tumors and / or unhealthy tissue within an organ from healthy tissue.

[0175] The control system 600 is configured to implement a hyperspectral imaging and visualization system in which molecular responses are utilized to detect and identify anatomical structures in the surgical field of view. The control system 600 includes conversion logic 648 to convert tissue data into information usable by the surgeon. For example, key structures in the anatomical structure can be identified with variable reflectivity based on wavelength relative to obscuring materials. In addition, the control system 600 combines the identified spectral features and structural light data in an image. For example, the control system 600 can be used to create a three-dimensional data set for use in a system with enhanced image overlays for surgical use. The additional visual information can be used with techniques both intraoperatively and preoperatively. In various cases, the control system 600 is configured to provide a warning to the clinician when in proximity to one or more key structures. Various algorithms can be employed to direct robotic automation and semi-automated methods based on the surgical procedure and proximity to key structures.

[0176] A projected light array is employed to determine tissue shape and motion intraoperatively. Alternatively, a flash lidar can be employed for surface mapping of the tissue.

[0177] The control system 600 is configured to detect key structures and provide image overlays of the key structures, and measure distances to the surface of visible tissue and to embedded / buried key structures. In other cases, the control system 600 can measure distances to the surface of visible tissue or detect key structures, and provide image overlays of the key structures.

[0178] The control system 600 includes a spectral control circuit 602. For example, the spectral control circuit 602 can be a field programmable gate array (FPGA) or another suitable circuit configuration as described herein in connection with Left lung The spectral control circuit 602 includes a processor 604 to receive video input signals from a video input processor 606. For example, the processor 604 can be configured for hyperspectral processing and can utilize C / C++ code. For example, the video input processor 606 receives video input of control (metadata) data such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signals from the video input processor 606 and provide video output signals to a video output processor 608 that includes, for example, hyperspectral video output that interfaces control (metadata) data. The video output processor 608 provides the video output signals to an image overlay controller 610.

[0179] The video input processor 606 is coupled to the camera 612 at the patient side via patient isolation circuitry 614. As previously described, the camera 612 includes a solid state image sensor 634. The patient isolation circuitry can include multiple transformers such that the patient is isolated from other circuitry in the system. The camera 612 receives intraoperative images through optics 632 and image sensor 634. The image sensor 634 can include, for example, a CMOS image sensor, or can include, for example, any of the image sensor technologies described herein in connection with Left upper lobe In one aspect, the camera 612 outputs images with a 14-bit / pixel signal. It will be appreciated that higher or lower pixel resolutions can be employed without departing from the scope of the present disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616 which processes the camera output signal 613 using a hardware register 618 and a Nios2 coprocessor 620. The color RGB fusion output signal is provided to the video input processor 606 and a laser pulse control circuit 622.

[0180] The laser pulse control circuit 622 controls a laser engine 624. The laser engine 624 outputs light at multiple wavelengths (λ1, λ2, λ3... λ n ) including near infrared (NIR). The laser engine 624 can operate in multiple modes. In one aspect, the laser engine 624 can operate in, for example, two modes. In a first mode (e.g., normal operating mode), the laser engine 624 outputs an illumination signal. In a second mode (e.g., identification mode), the laser engine 624 outputs RGBG and NIR light. In various instances, the laser engine 624 can operate in a polarized mode.

[0181] Light output 626 from the laser engine 624 illuminates a target anatomical structure in the intraoperative surgical site 627. The laser pulse control circuit 622 also controls a laser pulse controller 628 for a laser pattern projector 630 which projects a laser pattern 631 (such as a grid or pattern of lines and / or dots) of a predetermined wavelength (λ2) on the surgical tissue or organ at the surgical site 627. The camera 612 receives the patterned light as well as reflected light output through the camera optics 632. The image sensor 634 converts the received light into a digital signal.

[0182] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and the video input module 636 for reading laser patterns 631 projected by the laser pattern projector 630 onto a target anatomy at the surgical site 627. The processing module 638 processes the laser patterns 631 and outputs a first video output signal 640 representing distances to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing a three-dimensional rendered shape of tissue or organs of the target anatomy at the surgical site.

[0183] The first video output signal 640 and the second video output signal 642 include data representing the location of the critical structure on the three-dimensional surface model, which is provided to the integration module 643. In conjunction with data from the video output processor 608 of the spectral control circuit 602, the integration module 643 can determine a distance d A ( Segments of left upper lobe, major vessels / airways ) to the buried critical structure (e.g., via a triangulation algorithm 644), and the distance d A may be provided to the image overlay controller 610 via a video output processor 646. The above-described conversion logic can encompass conversion logic circuitry 648, an intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0184] In various instances, preoperative data 650 from CT or MRI scans can be employed to register or match certain three-dimensional deformable tissue. Such preoperative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610 so that such information can be overlaid with the view from the camera 612 and provided to the video monitor 652. The registration of preoperative data is further described in this document and in the aforementioned concurrently filed U.S. Patent Applications, including, for example, U.S. Patent Application No. 16 / 128,195, filed September 11, 2018, entitled “INTEGRATION OF IMAGING DATA,” which are incorporated by reference herein in their entireties.

[0185] The video monitor 652 can output the integrated / enhanced view from the image overlay controller 610. The clinician can select and / or toggle between different views on one or more monitors. On a first monitor 652a, the clinician can toggle between (A) a view in which a three-dimensional rendering of visible tissue is depicted and (B) an enhanced view in which one or more hidden critical structures are depicted on the three-dimensional rendering of visible tissue. On a second monitor 652b, the clinician can toggle, for example, distance measurements to the one or more hidden critical structures and / or the surface of the visible tissue.

[0186] The control system 600 and / or various control circuits thereof can be incorporated into the various surgical visualization systems disclosed herein.

[0187] Stomach A structured (or patterned) light system 700 is shown in accordance with at least one aspect of the present disclosure. As described herein, structured light in the form of stripes or lines, for example, can be projected from a light source and / or projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contours of the surface 705. The camera 720 of the imaging device 120 Stomach ) can be configured to detect the projected pattern of light on the surface 705, for example. The manner in which the projected pattern deforms as it strikes the surface 705 allows the vision system to calculate depth and surface information of the target anatomical structure.

[0188] In certain instances, invisible (or imperceptible) structured light can be utilized, where the structured light is used without interfering with other computer vision tasks that can be confused by the projected pattern. For example, infrared light or extremely fast visible light frame rates can be utilized that alternate between two completely opposite patterns to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0189] As described above, the various surgical visualization systems described herein can be used to visualize various different types of tissue and / or anatomical structures, including tissue and / or anatomical structures that can be obscured from visualization by EMR in the visible portion of the light spectrum. In one aspect, the surgical visualization systems can utilize a spectral imaging system to visualize different types of tissue based on different combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the tissue being visualized based on the absorption coefficients of the tissue at various EMR wavelengths. The spectral imaging system can further be configured to characterize the tissue type of the tissue being visualized based on the particular combination of constituent materials. To illustrate, Fundus, antrum, pylorus is a graph 2300 that depicts how the absorption coefficients of various biological materials vary across the EMR wavelength spectrum. In the graph 2300, the vertical axis 2303 represents the absorption coefficient of the biological material (e.g., in cm -1The horizontal axis 2304 represents the EMR wavelength (e.g., in μm). Graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of proteins at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxyhemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxyhemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials; therefore, tissue types visualized by a surgical visualization system can be identified and distinguished based on specific combinations of detected constituent materials. Thus, a spectral imaging system can be configured to emit multiple different wavelengths of EMR, determine the constituent materials of the tissue based on the absorbed EMR absorption response detected at different wavelengths, and then characterize the tissue type based on specific detection combinations of constituent materials.

[0190] His angle, angularis incisurae, greater curvature / lesser curvature This demonstrates the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures. FIG. 28 In this imaging system, a spectral emitter 2320 (e.g., a spectral light source 150) is used to visualize the surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissue and / or structures at the surgical site 2325 can be visualized by an image sensor 135. FIG. 28 The imaging system 142 receives data to visualize tissue and / or structures; these tissues and / or structures may be visible (e.g., located on the surface of surgical site 2325) or obscured (e.g., located below other tissues and / or structures at surgical site 2325). In this example, the imaging system 142 ( FIG. 30 The imaging system 142 can visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissues whose spectral characteristics do not conform to known or expected spectral characteristics) based on spectral features characterized by the different absorption properties (e.g., absorption coefficients) of the constituent materials of each of the different tissue / structure types. The visualized tissues and structures can be displayed on a display screen associated with or coupled to the imaging system 142, such as imaging system display 146. FIG. 30 ), Main display 2119 ( FIG. 30 ), non-sterile display 2109 ( FIG. 30 ), Hub Display 2215 ( FIG. 30 ), Device / Instrument Display 2237 ( FIG. 24 )wait.

[0191] Further, the imaging system 142 can be configured to customize or update the displayed surgical site visualization according to the identified tissue and / or structure types. For example, the imaging system 142 can display an edge 2330a on a display screen (e.g., the display 146) associated with the tumor 2332 being visualized. The edge 2330a can indicate an area or amount of tissue that should be resected to ensure complete resection of the tumor 2332. The control system 133 FIG. 31 ) can be configured to control or update the size of the edge 2330a based on the tissue and / or structures identified by the imaging system 142. In the illustrated example, the imaging system 142 has identified a number of abnormalities 2338 within the FOV. Accordingly, the control system 133 can adjust the displayed edge 2330a to a first updated edge 2330b having a size sufficient to encompass the abnormalities 2338. Further, the imaging system 142 has also identified an artery 2334 that partially overlaps with the initially displayed edge 2330a (as shown by the highlighted area 2336 of the artery 2334). Accordingly, the control system 133 can adjust the displayed edge 2330a to a second updated edge 2330c having a size sufficient to encompass the relevant portion of the artery 2334.

[0192] In addition to or instead of the absorption properties described above with respect to FIG. 2A and 13B , the tissues and / or structures can also be imaged or characterized over the EMR wavelength spectrum according to their reflective properties. For example, FIG. 2B various plots showing reflectivity of different types of tissues or structures at different EMR wavelengths. FIG. 2C is a graphical representation 1050 of exemplary ureter features relative to a mask. FIG. 32A-32D is a graphical representation 1052 of exemplary artery features relative to a mask. FIG. 32B is a graphical representation 1054 of exemplary nerve features relative to a mask. FIG. 32D The plots in represent the reflectivity as a function of wavelength (nm) of the respective reflectivity of the particular structures (ureter, artery, and nerve) relative to fat, lung tissue, and blood at the respective wavelengths. These plots are merely for illustrative purposes, and it should be understood that other tissues and / or structures can have respective detectable reflective features that would allow for identification and visualization of the tissues and / or structures.

[0193] In various scenarios, selected wavelengths for spectral imaging can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site (i.e., “selective spectral” imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized, enabling information to be acquired in real-time or near real-time and utilized during surgery. In various scenarios, the wavelength can be selected by the clinician or by control circuitry based on clinician input. In some cases, the wavelength can be selected based on, for example, machine learning and / or large datasets accessible to the control circuitry via the cloud.

[0194] The aforementioned application of spectral imaging to tissue can be used during surgery to measure the distance between a waveform transmitter and critical structures obscured by tissue. In one aspect of this disclosure, see now. FIG. 32A and FIG. 32B The diagram illustrates a time-of-flight sensor system 1104 utilizing waveforms 1124 and 1125. In some cases, the time-of-flight sensor system 1104 can be integrated into a surgical visualization system 100. FIG. 32B The time-of-flight sensor system 1104 includes a waveform transmitter 1106 and a waveform receiver 1108 on the same surgical device 1102. A transmitted wave 1124 extends from the transmitter 1106 to a critical structure 1101, and a received wave 1125 is reflected back from the critical structure 1101 by the receiver 1108. The surgical device 1102 is positioned through a cannula 1110 extending into a lumen 1107 in the patient.

[0195] Waveforms 1124 and 1125 are configured to penetrate the obscured tissue 1103. For example, the wavelengths of waveforms 1124 and 1125 may be in the NIR or SWIR wavelength spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal may be emitted from transmitter 1106 and may penetrate the tissue 1103 in which the critical structure 1101 is concealed. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various cases, waveforms 1124 and 1125 may be selected based on the spectral characteristics of the critical structure 1101 to target the critical structure 1101 within the tissue 1103, as further described herein. In various cases, transmitter 1106 is configured to provide binary signal on and off, such as FIG. 32C As shown, for example, this binary signal can be measured by receiver 1108.

[0196] Based on the delay between the transmitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d. FIG. 32D ). FIG. 33-35The time-of-flight timing diagram 1130 of the transmitter 1106 and receiver 1108 of the FIG. 2A d is a function of the distance d, and the distance d is given by:

[0197]

[0198] where:

[0199] c = the speed of light;

[0200] t = the length of the pulse;

[0201] qi = the charge accumulated while light is being emitted; and

[0202] q2 = the charge accumulated while light is not being emitted.

[0203] As provided herein, the time-of-flight of the waveforms 1124, 1125 corresponds to the distance d in FIG. 2B In various instances, the additional transmitter / receiver and / or the pulsed signal from the transmitter 1106 can be configured to emit a non-penetrating signal. The non-penetrating tissue can be configured to determine the distance from the transmitter to the surface 1105 of the obscuring tissue 1103. In various instances, the depth of the critical structure 1101 can be determined by:

[0204] d A = d w -d t .

[0205] where:

[0206] d A = the depth of the critical structure 1101;

[0207] d w = the distance from the transmitter 1106 to the critical structure 1101 (d in FIG. 2C

[0208] d t = the distance from the transmitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the obscuring tissue 1103.

[0209] In one aspect of the disclosure, referring now to FIG. 31 , a time-of-flight sensor system 1204 is shown that utilizes waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c. In certain instances, the time-of-flight sensor system 1204 can be incorporated into the surgical visualization system 100 FIG. 29 ​The time-of-flight sensor system 1204 includes a wave form transmitter 1206 and a wave form receiver 1208. The wave form transmitter 1206 is positioned on the first surgical device 1202a and the wave form receiver 1208 is positioned on the second surgical device 1202b. The surgical devices 1202a, 1202b are positioned through their respective trocar needles 1210a, 1210b, which extend into the patient’s cavity 1207. Transmit waves 1224a, 1224b, 1224c extend from the transmitter 1206 toward the surgical site, and receive waves 1225a, 1225b, 1225c reflect off various structures and / or surfaces at the surgical site back to the receiver 1208.

[0210] Different transmit waves 1224a, 1224b, 1224c are configured to target different types of material at the surgical site. For example, wave 1224a targets obscuring tissue 1203, wave 1224b targets a first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets a second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of the waves 1224a, 1224b, 1224c can be in the visible, NIR, or SWIR wavelength spectrum. For example, visible light can reflect off the surface 1205 of the tissue 1203, and NIR and / or SWIR wave forms can be configured to penetrate the surface 1205 of the tissue 1203. In various aspects, a spectroscopic signal (e.g., hyperspectral, multispectral, or selective spectral) or photoacoustic signal can be emitted from the transmitter 1206, as described herein. In various cases, the waves 1224b, 1224c can be selected to target the critical structures 1201a, 1201b within the tissue 1203 based on the spectroscopic characteristics of the critical structures 1201a, 1201b, as described further herein. Photoacoustic imaging is further described in various U.S. patent applications, which are incorporated by reference into the present disclosure.

[0211] The transmit waves 1224a, 1224b, 1224c can reflect off the target materials (i.e., the surface 1205, the first critical structure 1201a, and the second structure 1201b, respectively). The received wave forms 1225a, 1225b, 1225c can be delayed due to the distance d FIG. 34 The distance d 1a The distance d 2a The distance d 3a The distance d 1b The distance d 2b The distance d 2c The distance d

[0212] In a time-of-flight sensor system 1204 in which the transmitter 1206 and receiver 1208 can be independently positioned (e.g., positioned on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d can be calculated based on the known orientations of the transmitter 1206 and receiver 1208. 1a d 2a d 3a d 1b d 2b d 2c For example, when surgical devices 1202a and 1202b are controlled by a robot, these orientations can be known. Knowing the positions of transmitter 1206 and receiver 1208, the time it takes for the photon stream to target a tissue, and the information about that specific response received by receiver 1208 allows for the determination of distance d. 1a d 2a d 3a d 1b d 2b d 2c In one aspect, the distance to the shielded critical structures 1201a and 1201b can be triangulated using the transmitted wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0213] See still FIG. 35 In various situations, in the view provided to the clinician, receiver 1208 can be rotated such that the centroid of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such orientation can rapidly transmit one or more relevant distances and / or perspectives regarding key structures. For example, as... FIG. 34 As shown, the surgical site is displayed from a viewpoint perpendicular to the viewing plane (i.e., with blood vessels oriented in / outside the page) of the key structure 1201a. In various cases, this orientation may be the default setting; however, the view can be rotated or otherwise adjusted by the clinician. In some situations, the clinician may switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0214] In various instances, the receiver 1208 can be mounted on a trocar or cannula, such as trocar 1210b, through which the surgical device 1202b is positioned, for example. In other instances, the receiver 1208 can be mounted on a separate robotic arm whose three-dimensional position is known. In various instances, the receiver 1208 can be mounted on a movable arm separate from the robot that controls the surgical device 1202a, or can be mounted to an operating room (OR) table that can be registered to the robot coordinate plane intraoperatively. In such instances, the positions of the transmitter 1206 and receiver 1208 can be able to be registered to the same coordinate plane so that distances can be triangulated from the output from the time-of-flight sensor system 1204.

[0215] Combining a time-of-flight sensor system and near-infrared spectroscopy (NIRS), referred to as TOF-NIRS, which is capable of measuring the time-resolved signature of NIR light with nanosecond resolution, can be found in the article entitled "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT" (Journal of the American Society for Horticultural Science, May 2013, Vol. 138, No. 3, pp. 225-228), which is incorporated by reference herein in its entirety, and is available at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0216] In various cases, the time-of-flight spectral waveforms are configured to determine the depth of the critical structure and / or the proximity of the surgical device to the critical structure. Moreover, various surgical visualization systems disclosed herein include surface mapping logic configured to create a three-dimensional rendering of the surface of the visible tissue. In such cases, the clinician can know the proximity (or lack of proximity) of the surgical device to the critical structure even when the visible tissue obstructs the critical structure. In one case, the topography of the surgical site is provided on a monitor by the surface mapping logic. If the critical structure is near the surface of the tissue, the spectral imaging can communicate the location of the critical structure to the clinician. For example, the spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other cases, the spectral imaging can detect structures 10 mm or 20 mm below the surface of the tissue. Based on the known limits of the spectral imaging system, the system is configured to communicate that the critical structure is out of range if the spectral imaging system cannot detect the critical structure at all. Thus, the clinician can continue to move the surgical device and / or manipulate the tissue. When the critical structure moves within the range of the spectral imaging system, the system can identify the structure and thus communicate that the structure is in range. In such cases, an alert can be provided when the structure is initially identified and / or further moved within a predefined proximity zone. In such cases, the clinician can be provided with proximity information (i.e., lack of proximity) even when the spectral imaging system does not identify the critical structure with known boundaries / ranges.

[0217] Various surgical visualization systems disclosed herein can be configured to identify the presence of a critical structure and / or the proximity of the critical structure in a procedure and alert the clinician before the critical structure is damaged by inadvertent dissection and / or transection. In various aspects, the surgical visualization system is configured to identify one or more of the following critical structures: ureter, intestine, rectum, nerves (including phrenic nerve, recurrent laryngeal nerve [RLN], sacral promontory facial nerve, vagus nerve, and their branches), blood vessels (including pulmonary artery and lobar artery and pulmonary vein and lobar vein, inferior mesenteric artery [IMA] and its branches, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its branches (including middle colic artery, right colic artery, ileocolic artery), hepatic artery and its branches, portal vein and its branches, splenic artery / vein and its branches, external and internal (inferior epigastric) ileocolic vessels, short gastric arteries, uterine arteries, mid-sacral vessels, and lymph nodes. Moreover, the surgical visualization system is configured to indicate the proximity of the surgical device to the critical structure and / or alert the clinician when the surgical device is near the critical structure.

[0218] Various aspects of the present disclosure provide intraoperative critical structure identification (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein can include spectral imaging and surgical instrument tracking, which enables visualization of critical structures, for example, under the surface of tissue (such as 1.0 cm to 1.5 cm under the surface of tissue). In other cases, the surgical visualization system can identify structures less than 1.0 cm or greater than 1.5 cm under the surface of tissue. For example, a surgical visualization system that can identify structures only within 0.2 mm of the surface can be valuable even if the structures could not be seen due to depth. In various aspects, the surgical visualization system can augment the clinician’s view, for example, with a virtual depiction of the critical structure as a visible white light image superimposed on the surface of the visible tissue. The surgical visualization system can provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and can provide proximity alerts, for example, when the distal tip of the surgical instrument is moved within a certain range of the critical structure, such as within 1.0 cm of the critical structure.

[0219] Various surgical visualization systems disclosed herein can identify when dissection is too close to a critical structure. Dissection can be “too close” to a critical structure based on temperature (i.e., too hot near a critical structure that risks damaging / heating / melting the critical structure) and / or based on tension (i.e., too much tension near a critical structure that risks damaging / tearing / pulling the critical structure). For example, such surgical visualization systems can facilitate dissection around a blood vessel when the blood vessel is skeletonized prior to ligation. In various cases, a thermal imaging camera can be utilized to read the heat at the surgical site and provide a warning to the clinician based on the detected heat and distance from the tool to the structure. For example, if the temperature of the tool is above a predefined threshold, such as, for example, 120 °F, a proximity alert can be provided to the clinician at a first distance, such as, for example, 10 mm, and if the temperature of the tool is less than or equal to the predefined threshold, a proximity alert can be provided to the clinician at a second distance, such as, for example, 5 mm. The predefined threshold and / or warning distance can be a default setting and / or programmable by the clinician. Additionally or alternatively, the proximity alert can be associated with a thermal measurement made by the tool itself, such as a thermocouple that measures heat in the distal jaws of, for example, a monopolar or bipolar dissector or a vessel sealer.

[0220] The various surgical visualization systems disclosed herein can provide sufficient sensitivity regarding critical structures and specificity to enable a clinician to confidently proceed with a quick but safe dissection based on the standard of care and / or device safety data. The systems can function intraoperatively in real-time during a surgical procedure with minimal to no risk of ionizing radiation to the patient or clinician, and in various instances, no risk of ionizing radiation to the patient or clinician. Conversely, during fluoroscopy, the patient and clinician can be exposed to ionizing radiation via, for example, an x-ray beam used to observe anatomical structures in real-time.

[0221] The various surgical visualization systems disclosed herein can be configured to, for example, detect and identify one or more desired types of critical structures in a forward path of a surgical device, such as when the path of the surgical device is controlled by a robot. Additionally or alternatively, the surgical visualization systems can be configured to detect and identify one or more types of critical structures, for example, in a surrounding area and / or in multiple planes / dimensions of a surgical device.

[0222] The various surgical visualization systems disclosed herein can be easy to operate and / or interpret. Furthermore, the various surgical visualization systems can incorporate an “override” feature that allows a clinician to override the default settings and / or operation. For example, such as when the risk of a critical structure is less than the risk of avoiding the area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue can be greater than the risk of causing damage to the critical structure), a clinician can selectively turn off alerts from the surgical visualization system and / or get closer to a critical structure than the surgical visualization system suggests.

[0223] The various surgical visualization systems disclosed herein can be incorporated into a surgical system and / or used during a surgical procedure with limited impact on workflow. In other words, the implementation of the surgical visualization system can not change the way a surgical procedure is performed. Furthermore, the surgical visualization system can be more economical compared to the cost of an unintended transection. Data suggests that a reduction in unintended damage to critical structures can drive incremental reimbursement.

[0224] The various surgical visualization systems disclosed herein can operate in real-time or near real-time and far enough in advance to enable a clinician to anticipate a critical structure. For example, the surgical visualization systems can provide sufficient time to “slow down, assess, and avoid” in order to maximize the efficiency of a surgical procedure.

[0225] The various surgical visualization systems disclosed herein can not require contrast agents or dyes injected into tissue. For example, spectral imaging is configured to visualize hidden structures intraoperatively without the use of contrast agents or dyes. In other cases, contrast agents can be more easily injected into the appropriate tissue layer than other visualization systems. For example, the time between injection of a contrast agent and visualization of a critical structure can be less than two hours.

[0226] The various surgical visualization systems disclosed herein can be associated with clinical data and / or device data. For example, the data can provide a boundary as to how close an energy-enabled surgical device (or other potentially damaging device) should be from tissue that a surgeon does not want to damage. Any data module that interacts with the surgical visualization systems disclosed herein can be provided integrally with a robot or separately to enable use with independent surgical devices, for example, in open or laparoscopic surgery. In various cases, the surgical visualization systems can be compatible with robotic surgical systems. For example, the visualization images / information can be displayed in a robot console.

[0227] In various cases, a clinician can not know the location of a critical structure relative to a surgical tool. For example, when a critical structure is embedded in tissue, a clinician can not be able to determine the location of the critical structure. In certain cases, a clinician can want to keep a surgical device outside of a range of bearings around a critical structure and / or away from visible tissue that covers a hidden critical structure. When the bearings of a hidden critical structure are unknown, a clinician can be at risk of moving too close to the critical structure, which can result in inadvertent trauma and / or dissection of the critical structure and / or excessive energy, heat, and / or tension near the critical structure. Alternatively, a clinician can stay too far from a suspected location of a critical structure and be at risk of affecting tissue at a less ideal location in an attempt to avoid the critical structure.

[0228] The present disclosure provides a surgical visualization system that presents surgical device tracking relative to one or more critical structures. For example, the surgical visualization system can track the proximity of a surgical device relative to a critical structure. Such tracking can occur intraoperatively, in real-time, and / or near real-time. In various cases, the tracking data can be provided to a clinician via a display screen (e.g., monitor) of an imaging system.

[0229] In one aspect of the present disclosure, a surgical visualization system includes a surgical device including an emitter configured to emit a structured light pattern onto a visible surface, an imaging system including a camera configured to detect the structured light pattern embedded on the visible surface, and a control circuit in signal communication with the camera and the imaging system, wherein the control circuit is configured to determine a distance from the surgical device to an embedded structure and provide a signal to the imaging system indicative of the distance. For example, the distance can be determined by calculating a distance from the camera to a critical structure illuminated with a fluoroscopy technique and based on a three-dimensional view of the illuminated structure provided by images from multiple lenses of the camera (e.g., a left lens and a right lens). For example, the distance from the surgical device to the critical structure can be triangulated based on known positions of the surgical device and the camera. Alternative devices for determining a distance to an embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine a distance to a visible tissue that overlays / embeds a critical structure. For example, the surgical visualization system can identify a hidden critical structure and enhance a view of the hidden critical structure by depicting a schematic of the hidden critical structure (such as a line on a surface of the visible tissue) on the visible structure. The surgical visualization system can also determine a distance to the enhanced line on the visible tissue.

[0230] By providing a clinician with up-to-date information regarding the proximity of a surgical device to a hidden critical structure and / or a visible structure as provided by various surgical visualization systems as disclosed herein, the clinician can make a more informed decision regarding placement of the surgical device relative to the hidden critical structure. For example, the clinician can view the distance between the surgical device and the critical structure in real-time / intraoperatively and, in certain instances, an alert and / or warning can be provided by the imaging system when the surgical device is moved within a predefined proximity and / or region of the critical structure. In certain instances, an alert and / or warning can be provided when the trajectory of the surgical device indicates a possible collision with a “no-fly” zone near the critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more of the critical structure). In such instances, the clinician can maintain momentum throughout the surgical procedure without the clinician monitoring for suspect locations of the critical structure and the proximity of the surgical device thereto. As a result, certain surgical procedures can be performed more quickly, with fewer pauses / interruptions, and / or with improved accuracy and / or certainty. In one aspect, the surgical visualization system can be used to detect tissue variability, such as within an organ, to distinguish between tumor / cancerous / unhealthy tissue and healthy tissue. Such a surgical visualization system can maximize removal of unhealthy tissue while minimizing removal of healthy tissue.

[0231] FIG. 28

[0232] The various visualization or imaging systems described herein can be incorporated into a surgical hub system, as described in connection with FIG. 34 are shown and described in further detail below.

[0233] Referring to FIG. 34 The computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., the cloud 2104 can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which can include the remote server 2113. In one example, as shown in FIG. 34 The surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112, which are configured to communicate with one another and / or the hub 2106, as shown in

[0234] FIG. 35 An example of a surgical system 2102 for performing a surgical procedure on a patient who is lying on an operating table 2114 in a surgical operating room 2116 is shown. The robotic system 2110 is used as part of the surgical system 2102 in the surgical procedure. The robotic system 2110 includes a surgeon’s console 2118, a patient-side cart 2120 (surgical robot), and a surgical robot hub 2122. The patient-side cart 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision in the patient while the surgeon views the surgical site through the surgeon’s console 2118. An image of the surgical site can be obtained by a medical imaging device 2124, which can be manipulated by the patient-side cart 2120 to orient the imaging device 2124. The robot hub 2122 can be used to process the image of the surgical site for subsequent display to the surgeon through the surgeon’s console 2118.

[0235] Other types of robotic systems can be readily adapted for use with the surgical system 2102. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in various U.S. Patent Applications, which are incorporated by reference into the present disclosure.

[0236] Various examples of cloud-based analytics performed by the cloud 2104 and suitable for use with the present disclosure are described in various U.S. Patent Applications, which are incorporated by reference into the present disclosure.

[0237] In various aspects, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0238] The optical components of the imaging device 2124 can include one or more illumination sources and / or one or more lenses. The one or more illumination sources can be directed to illuminate portions of the surgical field. The one or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0239] The one or more illumination sources can be configured to radiate electromagnetic energy in the visible spectrum as well as the non-visible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or luminous spectrum) is that portion of the electromagnetic spectrum that is visible to (i.e., can be detected by) the human eye and can be referred to as visible light or simply light. A typical human eye will respond to wavelengths that are in the range of about 380 nm to about 750 nm in air.

[0240] The non-visible spectrum (i.e., the non-luminous spectrum) is that portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths shorter than about 380 nm and longer than about 750 nm). The non-visible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum and they become invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths shorter than about 380 nm are shorter than the violet spectrum and they become invisible ultraviolet, x-ray, and gamma ray electromagnetic radiation.

[0241] In various aspects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthoscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscope), endoscopes, laryngoscopes, nasopharyngorenoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0242] In one aspect, the imaging device employs multispectral monitoring to discern topography and underlying structures. A multispectral image is an image that captures image data within a specific range of wavelengths across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging can allow extraction of additional information that the human eye fails to capture with its receptors of red, green, and blue. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated by reference into the present disclosure. Multispectral monitoring can be a useful tool for repositioning the surgical field after completing a surgical task to perform one or more previously described tests on the treated tissue.

[0243] It is axiomatic that strict sterilization of the operating room and surgical equipment is required during any surgery. The strict hygiene and sterilization conditions required in a “surgical theater” (i.e., an operating or treatment room) require the highest possible sterility of all medical devices and equipment. Part of that sterilization process is the need to sterilize anything that comes into contact with the patient or penetrates the sterile field, including the imaging device 2124 and its attachments and components. It will be appreciated that the sterile field can be considered a designated area thought to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area surrounding a patient who has been readied for a surgical procedure. The sterile field can include scrubbed team members who are properly attired, as well as all equipment and fixtures in that area.

[0244] In various aspects, the visualization system 2108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays, which are strategically arranged relative to the sterile field as shown in FIG. 21 A. In one aspect, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. The various components of the visualization system 2108 are described in various U.S. Patent Applications, which are incorporated by reference into the present disclosure. FIG. 34

[0245] As shown in FIG. 21 A, the main display 2119 is positioned in the sterile field to be visible to the operator at the surgical table 2114. In addition, the visualization tower 21121 is positioned outside the sterile field. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 facing away from each other. The visualization system 2108, directed by the hub 2106, is configured to utilize the displays 2107, 2109, and 2119 to coordinate information flow to the operators inside and outside the sterile field. For example, the hub 2106 can cause the visualization system 2108 to display a snapshot of the surgical site recorded by the imaging device 2124 on the non-sterile display 2107 or 2109, while maintaining a real-time feed of the surgical site on the main display 2119. The snapshot on the non-sterile display 2107 or 2109 can allow the non-sterile operator to, for example, perform a diagnostic step related to the surgical procedure. FIG. 34

[0246] In one aspect, the hub 2106 is further configured to route diagnostic input or feedback entered by the non-sterile operator at the visualization tower 21121 to the main display 2119 inside the sterile field, where it can be viewed by the sterile operator at the operating table. In one example, the input can be a modified version of the snapshot displayed on the non-sterile display 2107 or 2109, which can be routed to the main display 2119 by the hub 2106.

[0247] Referring to​​FIG. 17-19 The surgical instrument 2112 is used in a surgical procedure as part of the surgical system 2102. The hub 2106 is also configured to coordinate a flow of information to a display of the surgical instrument 2112, as described in various U.S. Patent Applications incorporated by reference into the present disclosure. Diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 21121 can be routed by the hub 2106 to the surgical instrument display 2115 within the sterile field, where it can be observed by the operator of the surgical instrument 2112. Exemplary surgical instruments suitable for use with the surgical system 2102 are described in various U.S. Patent Applications incorporated by reference into the present disclosure.

[0248] FIG. 35 A computer-implemented interactive surgical system 2200 is shown. The computer- implemented interactive surgical system 2200 is similar in many respects to the computer- implemented interactive surgical system 2100. The surgical system 2200 includes at least one surgical hub 2236 in communication with a cloud 2204 that can include a remote server 2213. In one aspect, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 that connects to multiple operating theatre devices such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating theatre. The surgical hub 2236 includes a communication interface for communicably coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. As FIG. 35 As shown in the example, the surgical hub 2236 is coupled to an imaging module 2238 (which is coupled to an endoscope 2239), a generator module 2240 coupled to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communication module 2230, a processor module 2232, a storage array 2234, an intelligent device / instrument 2235 optionally coupled to a display 2237, and a non-contact sensor module 2242. The operating theatre devices are coupled to cloud computing resources and data storage via the surgical hub 2236. A robot hub 2222 can also be connected to the surgical hub 2236 and the cloud computing resources. The devices / instruments 2235, visualization systems 2209, etc. can be coupled to the surgical hub 2236 via wired or wireless communication standards or protocols, as described herein. The surgical hub 2236 can be coupled to a hub display 2215 (e.g., monitor, screen) to display and overlay images received from the imaging module, device / instrument displays, and / or other visualization systems 208. The hub display can also display data received from devices connected to the modular control tower in conjunction with images and overlay images.

[0249] FIG. 35

[0250] The various visualization systems or aspects of visualization systems described herein can be used as part of a situational awareness system that can be implemented or executed by the surgical hub 2106, 2236( FIG. 22 ) to provide situational awareness to the user. In particular, characterizing, identifying, and / or visualizing the surgical instruments or other surgical devices, including their positions, orientations, and actions, the tissue, the structures, the user, and other things located within the surgical scene or operating room can provide contextual data that the situational awareness system can utilize to infer the type of surgical procedure being performed or step thereof, the type of tissue and / or structure the surgeon is operating on, etc. The situational awareness system can then utilize this contextual data to provide alerts to the user, suggest to the user to take subsequent steps or actions, prepare surgical devices for their use (e.g., anticipate activation of an electrosurgical generator in anticipation of use of an electrosurgical instrument in a subsequent step of a surgical procedure), intelligently control surgical instruments (e.g., customize surgical instrument operating parameters based on the particular health of each patient), etc.

[0251] While "smart" devices that include control algorithms responsive to sensed data can be an improvement over "dumb" devices that operate without regard to sensed data, some sensed data can be incomplete or uncertain when considered in isolation, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the surgical context, e.g., knowing the type of tissue being operated on or the type of procedure being performed, a control algorithm can control a modular device incorrectly or sub-optimally given a particular context-less sensed data. The modular device can include any surgical device that can be controlled by a situational awareness system, such as a visualization system device (e.g., a camera or display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a smoke evacuator). For example, the optimal way for a control algorithm to control a surgical instrument in response to a particular sensed parameter can vary depending on the particular tissue type being operated on. This is due to the fact that different tissue types have different properties (e.g., tear resistance) and thus respond differently to actions taken by the surgical instrument. Thus, it can be desirable for the surgical instrument to take different actions even when the same measured value for a particular parameter is sensed. As one particular example, the optimal way for a surgical stapling and cutting instrument to be controlled in response to the instrument sensing an unexpectedly high force for closing its end effector will vary depending on whether the tissue type is prone to tearing or tear resistant. For tissue that is prone to tearing, such as lung tissue, the control algorithm for the instrument will optimally ramp down the motor speed in response to the unexpectedly high force for closing, thereby avoiding tearing the tissue. For tissue that is tear resistant, such as stomach tissue, the control algorithm for the instrument will optimally ramp up the motor speed in response to the unexpectedly high force for closing, thereby ensuring that the end effector is properly clamped on the tissue. Without knowing whether lung tissue or stomach tissue has been clamped, the control algorithm can make a sub-optimal decision.

[0252] One solution utilizes a surgical hub that includes a system configured to be able to derive information about a surgical procedure being performed based on data received from various data sources, and then control paired modular devices accordingly. In other words, the surgical hub is configured to infer information about a surgical procedure from received data, and then control modular devices paired with the surgical hub based on the inferred context of the surgical procedure. FIG. 36A-36CA diagram of a situationally aware surgical system 2400 is shown in accordance with at least one aspect of the present disclosure. In some examples, the data sources 2426 include, for example, the modular devices 2402 (which can include sensors configured to detect parameters associated with the patient and / or the modular devices themselves), a database 2422 (e.g., an EMR database containing patient records), and patient monitoring devices 2424 (e.g., a blood pressure (BP) monitor and an electrocardiography (EKG) monitor).

[0253] The surgical hub 2404 (which can be similar in many respects to the hub 106) can be configured to derive context information related to a surgical procedure from the data, for example, based on a particular combination of the received data or a particular order in which the data is received from the data sources 2426. The context information inferred from the received data can include, for example, a type of surgical procedure being performed, a particular step of the surgical procedure being performed by the surgeon, a type of tissue being operated on, or a body cavity that is the subject of the procedure. This ability of aspects of the surgical hub 2404 to derive or infer information related to a surgical procedure from received data can be referred to as “situationally aware.” In one example, the surgical hub 2404 can incorporate a situationally aware system, which is the hardware and / or programming associated with the surgical hub 2404 that derives context information related to a surgical procedure from received data.

[0254] The situationally aware system of the surgical hub 2404 can be configured to derive context information from the data received from the data sources 2426 in a variety of different ways. In one example, the situationally aware system includes a pattern recognition system or a machine learning system (e.g., an artificial neural network) that has been trained on training data to associate various inputs (e.g., data from the database 2422, the patient monitoring devices 2424, and / or the modular devices 2402) with corresponding context information about a surgical procedure. In other words, the machine learning system can be trained to accurately derive context information about a surgical procedure from provided inputs. In another example, the situationally aware system can include a lookup table that stores pre-characterized context information about a surgical procedure associated with one or more inputs (or ranges of inputs) that correspond to the context information. In response to a query with one or more inputs, the lookup table can return corresponding context information that the situationally aware system uses to control the modular devices 2402. In one example, the context information received by the situationally aware system of the surgical hub 2404 is associated with a particular control adjustment or a set of control adjustments for one or more modular devices 2402. In another example, the situationally aware system includes a further machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 2402 when provided with context information as input.

[0255] The surgical hub 2404, in conjunction with the situational awareness system, provides a number of benefits for the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which will in turn improve the precision of handling and / or use of data during the surgical procedure process. Returning to the previous example, the situational aware surgical hub 2404 can determine the type of tissue being operated on; thus, when an unexpectedly high force to close an end effector of a surgical instrument is detected, the situational aware surgical hub 2404 can correctly ramp up or ramp down the motor speed for the surgical instrument for the tissue type.

[0256] As another example, the type of tissue being operated on can affect the adjustment of compression rate and load threshold values for a surgical stapling and cutting instrument for a particular tissue gap measurement. The situational aware surgical hub 2404 can infer whether a surgical procedure being performed is a thoracic surgery or an abdominal surgery, allowing the surgical hub 2404 to determine whether the tissue clamped by an end effector of a surgical stapling and cutting instrument is lung tissue (for a thoracic surgery) or stomach tissue (for an abdominal surgery). The surgical hub 2404 can then adjust the compression rate and load threshold values of the surgical stapling and cutting instrument appropriately for the type of tissue.

[0257] As yet another example, the type of body cavity being operated in during an insufflation procedure can affect the functionality of a smoke evacuator. The situational aware surgical hub 2404 can determine whether a surgical site is under pressure (by determining that a surgical procedure is utilizing insufflation) and determine the type of surgery. As one type of surgery is typically performed within a particular body cavity, the surgical hub 2404 can then control the motor rate of the smoke evacuator appropriately for the body cavity in which it is operating. Thus, the situational aware surgical hub 2404 can provide consistent smoke evacuation for both thoracic and abdominal surgeries.

[0258] As yet another example, the type of procedure being performed can affect the optimal energy level at which an ultrasonic surgical instrument or a radiofrequency (RF) electrosurgical instrument operates. For example, arthroscopic procedures require a higher energy level because the end effector of the ultrasonic surgical instrument or the RF electrosurgical instrument is submerged in fluid. The situational awareness surgical hub 2404 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 can then adjust the RF power level or the ultrasonic amplitude of the generator (i.e., the "energy level") to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level at which an ultrasonic surgical instrument or a RF electrosurgical instrument operates. The situational awareness surgical hub 2404 can determine the type of surgical procedure being performed and then tailor the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument according to the expected tissue profile of that surgical procedure, respectively. Moreover, the situational awareness surgical hub 2404 can be configured to adjust the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument throughout the surgical procedure, rather than just on a procedure-by-procedure basis. The situational awareness surgical hub 2404 can determine the steps of the surgical procedure being performed or subsequently to be performed and then update the control algorithms for the generator and / or the ultrasonic surgical instrument or the RF electrosurgical instrument to set the energy level at a value appropriate for the expected tissue type according to that surgical procedure step.

[0259] As yet another example, data can be extracted from additional data sources 2426 to improve the conclusions drawn by the surgical hub 2404 from one data source 2426. The situational awareness surgical hub 2404 can augment the data it receives from the modular devices 2402 with contextual information about the surgical procedure that it has built from other data sources 2426. For example, the situational awareness surgical hub 2404 can be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. In some cases, however, the video or image data can be inconclusive. Accordingly, in one example, the surgical hub 2404 can be further configured to compare physiological measurements (e.g., blood pressure sensed by a BP monitor communicably connected to the surgical hub 2404) with the visual or image data of hemostasis (e.g., from a medical imaging device 124 FIG. 36A )) to determine the integrity of the suture or tissue weld. In other words, the situational awareness system of the surgical hub 2404 can consider physiological measurement data to provide additional context when analyzing visual data. The additional context can be useful when the visual data itself can be inconclusive or incomplete.

[0260] Another benefit includes proactively and automatically controlling paired modular devices 2402 according to the particular step of the surgical procedure being performed to reduce the number of times medical personnel need to interact with or control the surgical system 2400 during the course of the surgical procedure. For example, if the situational awareness surgical hub 2404 determines that a subsequent step of the procedure requires the use of an RF electrosurgical instrument, it can proactively activate the generator to which the instrument is connected. Proactively activating the energy source allows the instrument to be ready for use as soon as a previous step of the procedure is completed.

[0261] As another example, the situational awareness surgical hub 2404 can determine whether the current or subsequent step of the surgical procedure requires a different view or degree of magnification on the display according to the feature(s) the surgeon is expected to need to view at the surgical site. The surgical hub 2404 can then proactively change the view displayed (e.g., provided by a medical imaging device used for the visualization system 108) accordingly, such that the display is automatically adjusted throughout the surgical procedure.

[0262] As yet another example, the situational awareness surgical hub 2404 can determine which step of the surgical procedure is being performed or is next to be performed and whether that step of the surgical procedure requires particular data or comparisons between data. The surgical hub 2404 can be configured to automatically call up data screens based on the step of the surgical procedure being performed without waiting for the surgeon to request that particular information.

[0263] Another benefit includes checking for errors during setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 2404 can determine whether an operating room is set up correctly or optimally for a surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve the corresponding checklist, product locations, or setup requirements (e.g., from memory), and then compare the current operating room layout to the standard layout determined by the surgical hub 2404 for that type of surgical procedure being performed. In one exemplification, the surgical hub 2404 can be configured to compare a list of items for a procedure (e.g., scanned by a suitable scanner) and / or a list of devices paired with the surgical hub 2404 to a suggested or expected checklist of items and / or devices for a given surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing if there is any discontinuity between the lists. In one exemplification, the surgical hub 2404 can be configured to determine the relative distance or location of the modular devices 2402 and patient monitoring devices 2424, e.g., via proximity sensors. The surgical hub 2404 can compare the relative locations of the devices to a suggested or expected layout for a particular surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the suggested layout if there is any discontinuity between the layouts.

[0264] As another example, the situational awareness surgical hub 2404 can determine whether a surgeon (or other medical personnel) is making errors or otherwise deviating from an expected course of action during the course of a surgical procedure. For example, the surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve the corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the course of the surgical procedure to the expected steps or equipment determined by the surgical hub 2404 for that type of surgical procedure being performed. In one exemplification, the surgical hub 2404 can be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0265] In general, the situational awareness system for the surgical hub 2404 improves surgical procedure outcomes by adjusting surgical instruments (and other modular devices 2402) for the particular context of each surgical procedure, such as for different tissue types, and verifying actions during the surgical procedure. The situational awareness system also increases the efficiency of the surgeon performing the surgical procedure by automatically suggesting next steps, providing data, and adjusting displays and other modular devices 2402 in the operating room according to the particular context of the procedure.

[0266] Referring now to FIG. 25, FIG. 36B which shows a timeline 2500 depicting situational awareness of a hub, such as the surgical hub 106 or 206 FIG. 37 ) for a pulmonary segmentectomy procedure. The timeline 2500 is illustrative of a surgical procedure and the contextual information that the surgical hub 106, 206 can derive from data received from data sources at each step of the surgical procedure. The timeline 2500 depicts the typical steps that a nurse, surgeon, and other medical personnel will take during a pulmonary segmentectomy procedure, starting from setting up the operating room to transferring the patient to the post-anesthesia care unit.

[0267] The situational awareness surgical hub 106, 206 receives data from data sources throughout the surgical procedure, including data generated each time a medical personnel utilizes a modular device that is paired with the surgical hub 106, 206. The surgical hub 106, 206 can receive this data from the paired modular devices and other data sources, and continually derive inferences (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The situational awareness system of the surgical hub 106, 206 is able to, for example, record data related to the process for generating a report, verify the steps that medical personnel are taking, provide data or cues that can be relevant to a particular process step (e.g., via a display screen), adjust modular devices based on the context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and take any other such actions described above.

[0268] As a first step 2502 in this exemplary procedure, hospital staff retrieves the patient’s EMR from the hospital’s EMR database. Based on the selected patient data in the EMR, the surgical hub 106, 206 determines that the procedure to be performed is a thoracic procedure.

[0269] Second step 2504, staff scan incoming medical supplies for the procedure. The surgical hub 106, 206 cross-references the scanned supplies with a list of supplies utilized in various types of procedures and confirms that the mix of supplies corresponds to a thoracic procedure. Additionally, the surgical hub 106, 206 can also determine that the procedure is not a wedge procedure (as the incoming supplies lack certain supplies required for a thoracic wedge procedure, or otherwise do not correspond to a thoracic wedge procedure).

[0270] Third step 2506, medical personnel scan the patient band via a scanner communicably connected to the surgical hub 106, 206. The surgical hub 106, 206 can then confirm the identity of the patient based on the scanned data.

[0271] Fourth step 2508, medical staff turn on ancillary equipment. The ancillary equipment utilized can vary depending on the type of surgical procedure and the technology to be used by the surgeon, but in this illustrative case, they include a smoke evacuator, an insufflator, and a medical imaging device. When activated, as part of their initialization process, the ancillary equipment, as modular devices, can automatically pair with the surgical hub 106, 206 located in a particular vicinity of the modular devices. The surgical hub 106, 206 can then derive contextual information about the surgical procedure by detecting the types of modular devices that pair with it during this pre-procedural or initialization phase. In this particular example, the surgical hub 106, 206 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices. Based on the combination of data from the patient’s EMR, the list of medical supplies used in the procedure, and the types of modular devices connected to the hub, the surgical hub 106, 206 can generally infer the specific procedure that the surgical team will perform. Once the surgical hub 106, 206 knows what specific procedure is being performed, the surgical hub 106, 206 can retrieve the steps of that procedure from memory or the cloud and then cross-reference its subsequent receipt of data from connected data sources (e.g., the modular devices and patient monitoring devices) to infer what step of the surgical procedure the surgical team is performing.

[0272] Fifth step 2510, staff attach EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can pair with the surgical hub 106, 206. When the surgical hub 106, 206 begins to receive data from the patient monitoring devices, the surgical hub 106, 206 thus confirms that the patient is in the operating room.

[0273] At sixth step 2512, medical personnel induce anesthesia in the patient. The surgical hub 106, 206 can infer that the patient is under anesthesia based on data from modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Upon completion of the sixth step 2512, the pre-operative portion of the pulmonary segmentectomy procedure is complete, and the operative portion of the procedure begins.

[0274] At seventh step 2514, the lung of the patient being operated on is collapsed (while ventilation is switched to the contralateral lung). For example, the surgical hub 106, 206 can infer from ventilator data that the patient’s lung has been collapsed. The surgical hub 106, 206 can infer that the operative portion of the procedure has begun because it can compare the detection of the patient’s lung being collapsed to the expected steps of the procedure (which can have been previously accessed or retrieved), determining that collapsing the lung is the first surgical step in this particular procedure.

[0275] At eighth step 2516, a medical imaging device (e.g., an endoscope) is inserted, and video from the medical imaging device is initiated. The surgical hub 106, 206 receives medical imaging device data (i.e., video or image data) through its connection with the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 106, 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 106, 206 can determine that the particular procedure being performed is a segmentectomy, as opposed to a lobectomy (note that a wedge procedure has already been ruled out by the surgical hub 106, 206 based on data received at the second step 2504 of the procedure). From the medical imaging device 124 FIG. 2AThe data from the medical imaging devices can be used to determine situational information related to the type of procedure being performed in a number of different ways, including by determining the angle of the visualization orientation of the medical imaging devices relative to the patient anatomy, monitoring the number of medical imaging devices being utilized (i.e., activated and paired with the surgical hub 106, 206), and monitoring the types of visualization devices being utilized. For example, one technique for performing a VATS lobectomy places a camera in the lower anterior corner of the patient’s chest cavity above the diaphragm, while one technique for performing a VATS segmentectomy places a camera in an anterior intercostal position relative to the segmental fissure. Using pattern recognition or machine learning techniques, for example, the situational awareness system can be trained to recognize the positioning of the medical imaging devices from the visualization of the patient anatomy. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As another example, one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicably coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in a VATS lobectomy. By tracking any or all of this data from the medical imaging devices, the surgical hub 106, 206 can therefore determine the specific type of surgical procedure being performed and / or the technique being used for a particular type of surgical procedure.

[0276] At a ninth step 2518, the surgical team begins the dissection step of the procedure. The surgical hub 106, 206 can infer that the surgeon is dissecting to mobilize the patient’s lung because it receives data from the RF generator or the ultrasound generator indicating that an energy instrument is being fired. The surgical hub 106, 206 can cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being fired at that point in the procedure (i.e., after the completion of the previously-discussed steps of the procedure) corresponds to the dissection step. In certain instances, the energy instrument can be an energy tool mounted to a robotic arm of a robotic surgical system.

[0277] At a tenth step 2520, the surgical team continues with the stapling step of the procedure. The surgical hub 106, 206 can infer that the surgeon is stapling because it receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired. Similar to the previous steps, the surgical hub 106, 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the retrieved steps of the procedure. In certain instances, the surgical instrument can be a surgical tool mounted to a robotic arm of a robotic surgical system.

[0278] A tenth step 2522, a segmental resection portion of the procedure is performed. The surgical hub 106, 206 can infer that the surgeon is transecting soft tissue based on data from the surgical stapling and cutting instrument, including data from its cartridge. The cartridge data can correspond to, for example, the size or type of staple fired by the instrument. As different types of staples are used for different types of tissue, the cartridge data can be indicative of the type of tissue being stapled and / or transected. In this case, the type of staple fired is used for soft tissue (or other similar tissue type), which allows the surgical hub 106, 206 to infer that a segmental resection portion of the procedure is being performed.

[0279] In a twelfth step 2524, a node dissection step is performed. The surgical hub 106, 206 can infer that the surgical team is dissecting a node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. For this particular procedure, the RF or ultrasonic instrument utilized after transecting soft tissue corresponds to the node dissection step, which allows the surgical hub 106, 206 to make such an inference. It should be noted that the surgeon regularly switches back and forth between the surgical stapling / cutting instrument and the surgical energy (i.e., RF or ultrasonic) instrument depending on the specific step in the procedure, as different instruments are better suited for particular tasks. Thus, a particular sequence in which the stapling / cutting instrument and the surgical energy instrument are used can be indicative of the step of the procedure that the surgeon is performing. Further, in some cases, a robotic tool can be used for one or more steps in the surgical procedure, and / or a handheld surgical instrument can be used for one or more steps in the surgical procedure. One or more surgeons can, for example, alternate between the robotic tool and the handheld surgical instrument and / or can use both devices simultaneously. Upon completion of the twelfth step 2524, the incision is closed and the post-operative portion of the procedure begins.

[0280] A thirteenth step 2526, the patient is reversed from anesthesia. For example, the surgical hub 106, 206 can infer that the patient is waking up from anesthesia based on, for example, ventilator data (i.e., the patient’s respiratory rate begins to increase).

[0281] Finally, a fourteenth step 2528, the medical staff removes various patient monitoring devices from the patient. Thus, the surgical hub 2106, 2236 can infer that the patient is being transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices. As can be seen from the description of this example procedure, the surgical hub 2106, 2236 can determine or infer when each step of a given surgical procedure occurs from data received from various data sources that can be communicably coupled to the surgical hub 2106, 2236.

[0282] Situational awareness is further described in various U.S. Patent Applications incorporated by reference into the present disclosure, which is incorporated by reference herein. In certain instances, operation of a robotic surgical system, including for example the various robotic surgical systems disclosed herein, can be controlled by the hub 2106, 2236 based on its situational awareness and / or feedback from its components and / or based on information from the cloud 2104 FIG. 2B

[0283] FIG. 2C is a logic flow diagram of the process 4000 in accordance with at least one aspect of the present disclosure, which depicts a control program or logic configuration for associating visualization data with instrument data. The process 4000 is generally performed during a surgical procedure and includes receiving or deriving 4001, from a surgical visualization system, a first data set indicative of a visual aspect of a surgical instrument relative to a surgical field, i.e., visualization data, receiving or deriving 4002, from the surgical instrument, a second data set indicative of a functional aspect of the surgical instrument, i.e., instrument data, and associating 4003 the first data set with the second data set.

[0284] In at least one example, associating the visualization data with the instrument data is achieved by developing a composite data set from the visualization data and the instrument data. The process 4000 can further include comparing the composite data set to another composite data set, which can be received from an external source and / or can be derived from a previously collected composite data set. In at least one example, the process 4000 includes displaying a comparison of the two composite data sets, as described in greater detail below.

[0285] The visualization data of the process 4000 can be indicative of a visual aspect of an end effector of the surgical instrument relative to tissue in the surgical field. Additionally or alternatively, the visualization data can be indicative of a visual aspect of tissue treated by the end effector of the surgical instrument. In at least one example, the visualization data represents one or more positions of the end effector or a component thereof relative to tissue in the surgical field. Additionally or alternatively, the visualization data can represent one or more motions of the end effector or a component thereof relative to tissue in the surgical field. In at least one example, the visualization data represents one or more changes in shape, size, and / or color of tissue treated by the end effector of the surgical instrument.

[0286] In various aspects, the visualization data is derived from a surgical visualization system, e.g., the visualization systems 100, 160, 500, 2108. The visualization data can be derived from various measurements, readings, and / or any other suitable parameters monitored and / or captured by the surgical visualization system, as FIG. 36B ​More detail is described. In various examples, the visualization data is indicative of one or more visual aspects of tissue in the surgical field and / or one or more visual aspects of the surgical instrument relative to the tissue in the surgical field. In certain examples, the visualization data represents or identifies a position and / or motion of an end effector of the surgical instrument relative to a key structure in the surgical field (e.g., key structure 101 in FIG. 1). In certain examples, the visualization data is derived from surface mapping data, imaging data, tissue identification data, and / or distance data computed by the surface mapping logic 136, the imaging logic 138, the tissue identification logic 140, or the distance determination logic 141, or any combination of the logics 136, 138, 140, and 141. FIG. 36B

[0287] In at least one example, the visualization data is derived from tissue identification and geometric surface mapping performed by the visualization system 100 in combination with the distance sensor system 104, as described in connection with FIG. 36C More detail is described. In at least one example, the visualization data is derived from measurements, readings, or any other sensor data captured by the imaging device 120. As described in connection with FIG. 36C The imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths, as described in connection with

[0288] Additionally or alternatively, the visualization data can be derived from measurements, readings, or any suitable sensor data captured by any suitable imaging device, including, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or non-visible), and structured light patterns (visible or non-visible). In at least one example, the visualization data is derived from the visualization system 160, which includes the optical waveform emitter 123 and the waveform sensor 122 configured to detect reflected waveforms, as described in connection with FIG. 38 and FIG. 2A More detail is described. In yet another example, the visualization data is derived from a visualization system that includes a three-dimensional (3D) camera and associated electronic processing circuitry, such as the visualization system 500. In yet another example, the visualization data is derived from the structured (or patterned) light system 700 in combination with FIG. 2B More detail is described. The foregoing examples can be used individually or in combination to derive the visualization data of the process 4000.

[0289] ​The instrument data of the process 4000 can be indicative of one or more operations of one or more internal components of a surgical instrument. In at least one example, the instrument data represents one or more operational parameters of the internal components of the surgical instrument. The instrument data can represent one or more positions and / or one or more motions of one or more internal components of the surgical instrument. In at least one example, the internal component is a cutting member configured to cut tissue during a firing sequence of the surgical instrument. Additionally or alternatively, the internal component can include one or more staples configured to be fired into tissue during a firing sequence of the surgical instrument.

[0290] In at least one example, the instrument data represents one or more operations of one or more components of one or more drive assemblies (e.g., an articulation drive assembly, a closure drive assembly, a rotation drive assembly, and / or a firing drive assembly) of the surgical instrument. In at least one example, the instrument data set represents one or more operations of one or more drive members (e.g., an articulation drive member, a closure drive member, a rotation drive member, and / or a firing drive member) of the surgical instrument.

[0291] FIG. 2C is a schematic view of an example surgical instrument 4600 for use with the process 4000, which is similar in many respects to other surgical instruments or tools described in the present disclosure (e.g., the surgical instrument 2112). For the sake of brevity, the present disclosure describes various aspects of the process 4000 using only handheld surgical instruments. However, this is not limiting. Such aspects of the process 4000 can equally be implemented using robotic surgical tools (e.g., the surgical tool 2117).

[0292] The surgical instrument 4600 includes a plurality of motors that can be activated to perform various functions. The plurality of motors of the surgical instrument 4600 can be activated to cause a firing motion, a closure motion, and / or an articulation motion in an end effector. The firing motion, the closure motion, and / or the articulation motion can be communicated to an end effector of the surgical instrument 4600, for example, by a shaft assembly. However, in other examples, a surgical instrument for use with the process 4000 can be configured to manually perform one or more of the firing motion, the closure motion, and the articulation motion. In at least one example, the surgical instrument 4600 includes an end effector that treats tissue by deploying staples into the tissue. In another example, the surgical instrument 4600 includes an end effector that treats tissue by applying a treatment energy to the tissue.

[0293] In some cases, the surgical instrument 4600 includes a firing motor 4602. The firing motor 4602 may be operatively coupled to a firing motor drive assembly 4604, which may be configured to transmit the firing motion generated by the firing motor 4602 to an end effector, specifically for moving a firing member in the form of an I-beam, which may include a cutting member, for example. In some cases, the firing motion generated by the firing motor 4602 may cause, for example, a staple to be deployed from a staple cartridge into tissue captured by the end effector, and optionally, cause the cutting member of the I-beam to be advanced to cut the captured tissue.

[0294] In some cases, surgical instruments or tools may include a closure motor 4603. The closure motor 4603 may be operatively coupled to a closure motor drive assembly 4605, which is configured to transmit the closing motion generated by the closure motor 4603 to an end effector, specifically for displacing the closure tube to close the anvil and compress tissue between the anvil and the cartridge. The closing motion may cause, for example, the end effector to change from an open configuration to an approach configuration to capture tissue.

[0295] In some cases, surgical instruments or tools may include, for example, one or more articulated motors 4606a, 4606b. The articulated motors 4606a, 4606b may be operatively coupled to corresponding articulated motor drive assemblies 4608a, 4608b, which may be configured to transmit joint motion generated by the articulated motors 4606a, 4606b to an end effector. In some cases, the joint motion may cause the end effector to articulate relative to an axis, for example.

[0296] In some cases, surgical instruments or tools may include a control module 4610 that can be used with multiple motors of the surgical instrument 4600. Each of motors 4602, 4603, 4606a, and 4606b may include a torque sensor to measure the output torque on the motor shaft. Forces on the end effector can be sensed in any conventional manner, such as by a force sensor on the outside of the jaws or by a torque sensor for the motor used to actuate the jaws.

[0297] In various situations, such as FIG. 36C As shown, control module 4610 may include motor driver 4626, which may include one or more H-bridge FETs. Motor driver 4626 may, for example, modulate the power delivered from power source 4628 to the motor coupled to control module 4610 based on input from microcontroller 4620 (“controller”). In some cases, when the motor is coupled to control module 4610, controller 4620 may be used, for example, to determine the current consumed by the motor, as described above.

[0298] In some cases, controller 4620 may include microprocessor 4622 (“processor”) and one or more non-transitory computer-readable medium or memory units 4624 (“memory”). In some cases, memory 4624 may store various program instructions that, when executed, cause processor 4622 to perform the various functions and / or calculations described herein. In some cases, one or more memory units in memory unit 4624 may be coupled to processor 4622, for example. In various cases, processor 4622 may control motor driver 4626 to control the position, direction of rotation, and / or speed of a motor coupled to control module 4610.

[0299] In some cases, one or more mechanisms and / or sensors (e.g., sensor 4630) may be configured to detect the force (closing force "FTC") applied by the jaws of the end effector of surgical instrument 4600 to tissue captured between the jaws. The FTC may be transmitted to the jaws of the end effector via the closing motor drive assembly 4605. Alternatively or additionally, sensor 4630 may be configured to detect the force (firing force "FTF") applied to the end effector via the firing motor drive assembly 4604. In various examples, sensor 4630 may be configured to sense closing actuation (e.g., motor current and FTC), firing actuation (e.g., motor current and FTF), joint movement (e.g., angular position of the end effector), and rotation of the shaft or end effector.

[0300] One or more aspects of process 4000 may be executed by one or more control circuits of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4000 are executed by control circuits (e.g., FIG. 36C The control circuit 4000 executes the process, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4000. Alternatively or additionally, one or more aspects of the process 4000 may be executed by combinational logic circuitry (e.g., FIG. 39 The control circuit 410) and / or sequential logic circuit (e.g., FIG. 2A The process 4000 is executed by the control circuit 420. Furthermore, the process 4000 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0301] In all aspects, the process 4000 can be implemented via a computer-based interactive surgical system 2100.FIG. 2B implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., the cloud 2104 can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which can include the remote server 2113. The control circuit executing one or more aspects of the process 4000 can be a component of a visualization system (e.g., the visualization systems 100, 160, 500, 2108) and can be in communication with a surgical instrument (e.g., the surgical instruments 2112, 4600) to receive instrument data therefrom. Communication between the surgical instrument and the control circuit of the visualization system can be direct communication, or the instrument data can be routed through, for example, the surgical hub 2106, to the visualization system. In at least one example, the control circuit executing one or more aspects of the process 4000 can be a component of the surgical hub 2106.

[0302] Referring to FIG. 41, in various examples, the visualization data 4010 is associated with the instrument data 4011 by developing a composite data set 4012 from the visualization data 4010 and the instrument data 4011. FIG. 2C In various examples, the current user’s composite data set 4012 is displayed in a graph 4013 developed from the current user’s visualization data 4010 and the current user’s instrument data 4011. FIG. 19 The graph 4013 illustrates the visualization data 4010 representing a first use cycle of the surgical instrument 4600 involving jaw positioning, clamping, and firing of the surgical instrument 4600. The graph 4013 also depicts the visualization data 4010 representing the start of a second use cycle of the surgical instrument 4600 in which the jaws are repositioned for a second clamping and firing of the surgical instrument 4600. The graph 4013 further depicts the current user’s instrument data 4011 in the form of FTC data 4014 associated with the clamping visualization data and FTF data 4015 associated with the firing visualization data.

[0303] As described above, the visualization data 4010 is derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can represent, for example, a distance between an end effector of the surgical instrument 4600 and a critical structure in a surgical field of view during positioning, clamping, and / or firing of the end effector of the surgical instrument 4600. In at least one example, the visualization system identifies the end effector or a component thereof in the surgical field of view, identifies a critical structure in the surgical field of view, and tracks a position of the end effector or the component thereof relative to the critical structure or relative to tissue surrounding the critical structure. In at least one example, the visualization system identifies a jaw of the end effector in the surgical field of view, identifies a critical structure in the surgical field of view, and tracks a position of the jaw relative to the critical structure or relative to tissue surrounding the critical structure in surgery.

[0304] In at least one example, the critical structure is a tumor. To remove the tumor, a surgeon often prefers to cut tissue along a safety margin around the tumor to ensure that the entire tumor is removed. In such examples, the visualization data 4010 can represent a distance between a jaw of the end effector of the surgical instrument 4600 and the safety margin of the tumor during positioning, clamping, and / or firing of the surgical instrument 4600.

[0305] The process 4000 can further include comparing the current user’s composite dataset 4012 to another composite dataset 4012’ that can be received from an external source and / or that can be derived from a previously collected composite dataset. The plot 4013 illustrates a comparison between the current user’s composite dataset 4012 and another composite dataset 4012’ that includes visualization data 4010’ and instrument data 4011’ including FTC data 4014’ and FTF data 4015’. The comparison can be presented to the user of the surgical instrument 4600 in real-time in the form of the plot 4013 or in any other suitable format. The control circuit executing one or more aspects of the process 4000 can cause the comparison of the two composite datasets to be displayed on any suitable screen within the operating room (e.g., a screen of the visualization system). In at least one example, the comparison can be displayed with real-time video of the surgical field of view captured on any suitable screen within the operating room. In at least one example, the control circuit is configured to adjust the instrument parameters to address a detected deviation between the first and second composite datasets.

[0306] Further, the control circuit executing one or more aspects of the process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can further cause the current state of the instrument data (e.g., FTF data and / or FTC data) to be displayed relative to a best practice equivalent. In at least one example, the control circuit is configured to adjust the instrument parameters to address a detected deviation between the current state of the instrument data and the best practice equivalent. FIG. 41AIn the illustrated example, the current value of the FTC - represented by circle 4020 - is depicted in real-time against a gauge 4021 with an indicator 4022 representing the best practice FTC. Likewise, the current value of the FTF - represented by circle 4023 - is depicted against a gauge 4024 with an indicator 4025 representing the best practice FTF. Such information can be overlaid in real-time on a video feed of the surgical field of view.

[0307] FIG. 1 The illustrated example warns the user that the current current FTC is higher than the best practice FTC, and that the current FTF is also higher than the best practice FTF. If the current value of the FTF and / or the FTC reaches and / or moves beyond a predetermined threshold, the control circuit executing one or more aspects of the process 4000 can further warn the current user of the surgical instrument 4600 using an audible, visual, and / or tactile warning mechanism.

[0308] In certain instances, the control circuit executing one or more aspects of the process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can further provide projected instrument data to the current user of the surgical instrument 4600 based on the current investment data. For example, as FIG. 13A-13E As illustrated, a projected FTF 4015” is determined based on the current value of the FTF, and further displayed on the graph 4013 against the current FTF 4015 and the previously collected FTF’. Additionally or alternatively, the projected FTF circle 4026 can be displayed against the gauge 4024, as FIG. 29-38 As illustrated.

[0309] In various aspects, the previously collected composite dataset and / or the best practice FTF and / or the FTC are determined from previous uses of the surgical instrument 4600 in the same surgical procedure and / or other surgical procedures performed by the user, other users within the hospital, and / or users in other hospitals. Such data can be imported from the cloud 104, for example, to the control circuit executing one or more aspects of the process 4000 (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620).

[0310] In various aspects, the control circuit executing one or more aspects of the process 4000 (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can cause feedback measurements of tissue thickness, compression, and stiffness to be visually overlaid on a screen displaying a real-time feed of the surgical instrument 4600 in the surgical field as the jaws of the end effector begin to deform the tissue captured therebetween during the clamping phase. The visual overlay correlates the visualizations representing tissue deformation with the change in clamping force over time. This correlation helps the user confirm the correct cartridge selection, determine the time to start firing, and determine the appropriate firing speed. This correlation can further inform an adaptive clamping algorithm. The adaptive firing speed changes can be informed by the measured force and changes in tissue motion proximate to the jaws of the surgical instrument 4600 (e.g., principal strain, tissue slippage, etc.), with gauges or meters conveying the results overlaid on the screen displaying the real-time feed of the end effector in the surgical field.

[0311] In addition to the above, the kinematics of the surgical instrument 4600 can further indicate instrument operation relative to another use or user. The kinematics can be determined via accelerometers, torque sensors, force sensors, motor encoders, or any other suitable sensors, and can yield various force, velocity, and / or acceleration data of the surgical instrument or components thereof for correlation with corresponding visualization data.

[0312] In various aspects, if deviations from best practice surgical techniques are detected from the visualization data and / or instrument data, the control circuit executing one or more aspects of the process 4000 (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can cause an alternative surgical technique to be presented. In at least one example, an adaptive display of instrument motion, forces, tissue impedance, and results from the recommended alternative technique is presented. In the case where the visualization data indicates that a blood vessel and clip applier are detected in the surgical field, the control circuit executing one or more aspects of the process 4000 (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can further ensure perpendicularity of the blood vessel relative to the clip applier. The control circuit can suggest changes to position, orientation, and / or roll angle to achieve the desired perpendicularity.

[0313] In various aspects, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can perform the process 4000 by comparing real-time visualization data to a pre-operative plan simulation. A user can simulate a surgical procedure with a pre-operative patient scan. The pre-operative plan simulation can allow the user to follow a specific pre-operative plan based on training. The control circuit can be configured to correlate fiducials of the pre-operative scan / simulation to current visualization data. In at least one example, the control circuit can employ boundary tracking of objects to establish the correlation.

[0314] As a surgical instrument interacts with tissue and deforms the surface geometry, the change in surface geometry with position of the surgical instrument can be computed. For a given change in position of the surgical instrument while in contact with tissue, the corresponding change in tissue geometry can depend on the sub-surface structures in the region of tissue contacted by the surgical instrument. For example, in a thoracic surgery, the change in tissue geometry in a region with airway sub-structures differs from a region with parenchymal sub-structures. Generally, harder sub-structures produce a smaller change in surface tissue geometry in response to a given change in position of the surgical instrument. In various aspects, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to compute a running average of the change in surface geometry with change in position of the surgical instrument for a given patient, giving rise to patient-specific differences. Additionally or alternatively, the computed running average can be compared to a second set of previously collected data. In certain instances, a surface reference can be selected when no change in surface geometry is measured for each change in tool position. In at least one example, the control circuit can be configured to determine the location of sub-structures based on the change in surface geometry detected in response to a given contact between a region of tissue and the surgical instrument.

[0315] Further, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to maintain a set instrument-tissue contact throughout a tissue treatment based on a correlation between the set instrument-tissue contact and one or more changes in tissue surface geometry associated with the set instrument-tissue contact. For example, the end effector of the surgical instrument 4600 can set a desired compression of the instrument-tissue contact to clamp tissue between its jaws. A corresponding change in tissue surface geometry can be detected by the visualization system. Further, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can derive visualization data indicative of the change in tissue surface geometry associated with the desired compression. The control circuit can further cause the closure motor 4603 ​The motor settings of the surgical instrument 4600 are automatically adjusted to maintain the change in tissue surface geometry associated with the desired compression. This arrangement requires continuous interaction between the surgical instrument 4600 and the visualization system to maintain the change in tissue surface geometry associated with the desired compression by continuously adjusting the compression of the tissue by the jaws based on the visualization data.

[0316] In yet another example, where the surgical instrument 4600 is a robotic tool attached to a robotic arm of a robotic surgical system (e.g., the robotic system 110), the robotic surgical system can be configured to automatically adjust one or more components of the robotic surgical system to maintain set surface contact with the tissue based on the visualization data derived from the change in tissue surface geometry detected in response to the set surface contact with the tissue.

[0317] In various examples, the visualization data can be used in conjunction with measured instrument data to maintain contact between tissues in position or load control, allowing the user to manipulate the tissues to apply a predefined load to the tissues as the instrument is moved relative to the tissues. The user can specify that they want to maintain contact or maintain pressure, and the visual tracking of the instrument, as well as the internal load of the instrument, can be used to achieve repositioning without changing fixed parameters.

[0318] Referring to ​ and ​ The screen 4601 of the visualization system (e.g., the visualization systems 100, 160, 500, 2108) displays a real-time video feed of the surgical field during a surgical procedure. For example, the end effector 4642 of the surgical instrument 4600 includes jaws that clamp tissue near a tumor identified in the surgical field via superimposed MRI images. The jaws of the end effector 4642 include an anvil 4643 and a channel that houses a staple cartridge. At least one of the anvil 4643 and the channel is movable relative to the other to capture tissue between the anvil 4643 and the staple cartridge. The captured tissue is then stapled via staples 4644 that are deployable from the staple cartridge during a firing sequence of the surgical instrument 4600. In addition, the captured tissue is cut via a cutting member 4645 that is advanced distally during the firing sequence but lags slightly behind the deployment of the staples.

[0319] As ​As will be apparent, the position and / or motion of the captured tissue and certain internal components of the end effector 4642, such as the staples 4644 and the cutting member 4645, can not be visible in the ordinary view 4640 of the real-time feed on the screen 4601 during the firing sequence. Certain end effectors include windows 4641, 4653 that allow a partial view of the cutting member 4645 at the beginning and end of the firing sequence, but not during the firing sequence. Thus, the user of the surgical instrument 4600 cannot track the firing sequence progress on the screen 4601.

[0320] ​ A logic flow diagram of a process 4030 in accordance with at least one aspect of the present disclosure depicts a control program or logic configuration that synchronizes the motion of a virtual representation of an end effector component with the actual motion of the end effector component. The process 4030 is generally performed during a surgical procedure and includes detecting 4031 the motion of an internal component of an end effector during a firing sequence, for example by superimposing 4032 a virtual representation of the internal component on the end effector for presentation, and synchronizing 4033 the motion of the virtual representation on the screen 4601 with the detected motion of the internal component.

[0321] One or more aspects of the process 4030 can be performed by one or more of the control circuits described in the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4030 are performed by a control circuit (e.g., control circuit 400) that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4030. Additionally or alternatively, one or more aspects of the process 4030 can be performed by a combinational logic circuit (e.g., control circuit 410) and / or a sequential logic circuit (e.g., control circuit 420). Moreover, the process 4030 can be performed by any suitable circuit having any suitable hardware and / or software components that can be located in or associated with various suitable systems described in the present disclosure. ​ ​ ​

[0322] ​​​In various examples, a control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4030 can receive instrument data indicative of the motion of the internal components of end effector 4642 during its firing sequence. For example, the motion of the internal components can be tracked using a conventional rotary encoder of firing motor 4602. In other examples, the motion of the internal components can be tracked by a tracking system employing an absolute positioning system. A detailed description of an absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” published October 19, 2017, which is incorporated by reference herein in its entirety. In certain examples, the motion of the internal components can be tracked using one or more position sensors, which can include any number of magnetic sensing elements, such as magnetic sensors classified according to whether they measure the total magnetic field or the vector components of the magnetic field.

[0323] In various aspects, process 4030 includes a superimposition trigger. In at least one example, the superimposition trigger can detect that tissue is captured by end effector 4642. If tissue captured by end effector 4642 is detected, process 4030 superimposes a virtual representation of cutting member 4645 in the starting position onto end effector 4642. Process 4030 further includes projecting staple lines, outlining where staples will be deployed into the captured tissue. Additionally, in response to a user activation of a firing sequence, process 4030 causes the virtual representation of cutting member 4645 to move distally, simulating the actual motion of cutting member 4645 within end effector 4642. As the staples are deployed, process 4030 converts un-fired staples to fired staples, allowing the user to track staple deployment and advancement of cutting member 4645 in real-time visually.

[0324] In various examples, for example, a control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4030 can determine a position of a cutting member (e.g., cutting member 4645) of an end effector (e.g., end effector 4642) based on instrument data indicative of a position of a firing member (e.g., firing member 4604) of a firing motor (e.g., firing motor 4602) of a surgical instrument (e.g., surgical instrument 4600) during a firing sequence of the surgical instrument. ​Tissue capture by the end effector 4642 is detected by instrument data of the force applied to the jaws of the end effector 4642 by the closing motor drive assembly 4605. The control circuitry can further determine the position of the end effector within the surgical field of view based on visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108). In at least one example, the end effector position can be determined relative to a reference point in the tissue (e.g., a critical structure).

[0325] In any case, the control circuitry causes a virtual representation of the internal components to be superimposed on the end effector 4642 on screen 4601 at a position commensurate with the position of the internal components within the end effector. The control circuitry further causes the projected virtual representation of the internal components to move synchronously with the internal components during the firing sequence. In at least one example, synchronization is improved by incorporating markers on the end effector 4642, which the control circuitry can use as reference points to determine where to superimpose the virtual representation.

[0326] ​ An enhanced view 4651 shows a real-time feed of the surgical field of view on screen 4601. ​ In the example of the enhanced view 4651, virtual representations of pins 4644 and cutting members 4645 are superimposed on the end effector 4642 during the firing sequence. The superimposition distinguishes fired pins 4644a from unfired pins 4644b and complete cutting lines 4646a from projected cutting lines 4646b that track the progress of the firing sequence. The superimposition further shows the starting point of pin lines 4647 and the projected ends 4649 of pin lines that do not reach the tissue end. Furthermore, based on the superimposition of the tumor MRI image, a safe margin distance “d” between the tumor and the projected cutting lines 4646b is measured and presented along with the superimposition. The superimposed safe margin distance “d” assures the user that all tumor tissue will be removed.

[0327] like ​ As shown, the control circuitry is configured to enable the visualization system to continuously reposition the virtual representation of internal components in relation to the actual movement of those components. ​ In the example, the overlay shows the completed cutting line 4646 slightly behind the fired pin line 4644a by a distance "d1", which assures the user that the firing sequence is proceeding correctly.

[0328] Now for reference ​The visualization system (e.g., visualization systems 100, 160, 500, 2108) can employ a tool illumination 4058 and a camera 4059 to detect and / or define trocar position. Based on the determined trocar position, the user can be guided to the most appropriate trocar port to complete the intended function based on time efficiency, location of critical structures, and / or to avoid or risk.

[0329] ​ Three trocar positions (trocar 1, trocar 2, trocar 3) are shown extending through the body wall 4050 at different locations and orientations relative to the body wall and relative to a critical structure 4051 in a cavity 4052 within the body wall 4050. The trocars are represented by arrows 4054, 4055, 4056. The illumination tool 4058 can be used to detect the trocar position by using cascading light or images of the surrounding environment. Additionally, the light source of the illumination tool 4058 can be a rotatable light source. In at least one example, the light source of the illumination tool 4058 and the camera 4059 are used to detect the distance of the trocar relative to a target location (e.g., the critical structure 4051). In various examples, the visualization system can suggest a change in position of the instrument if a more optimal instrument position is determined based on the visualization data resulting from the camera 4059 recording the light projected by the light source of the illumination tool 4058. A screen 4060 can display the distance between the trocar and the target tissue, whether tool access through the trocar is acceptable, the risks associated with utilizing the trocar, and / or the expected operation time using the trocar, which can help the user select the optimal trocar for introducing a surgical tool into the cavity 4052.

[0330] In various aspects, the surgical hub (e.g., surgical hub 2106, 2122) can suggest the optimal trocar for inserting a surgical tool into the cavity 4052 based on, for example, user characteristics that can be received from a user database. The user characteristics include user hand dominance, patient side user preference, and / or user physical characteristics (e.g., height, arm length, range of motion). The surgical hub can utilize these characteristics of the available trocars, position and orientation data, and / or location data of critical structures to select the optimal trocar for inserting a surgical tool in an effort to reduce user fatigue and improve efficiency. In various aspects, the surgical hub can further cause the surgical instrument to invert its control if the user inverts the end effector orientation.

[0331] The surgical hub can reconfigure the output of the surgical instrument based on the visualization data. For example, if the visualization data indicates that the surgical instrument is retracting or is being used to perform a different task, the surgical hub can inhibit activation of the therapeutic energy output of the surgical instrument.

[0332] In various aspects, the visualization system can be configured to track a blood surface or estimate a blood volume based on reflected IR or red light wavelengths to delineate blood from non-blood surfaces and surface geometry measurements. This can be reported as an absolute static measurement or rate of change to provide quantitative data about the amount and degree of change of bleeding.

[0333] Reference is made to ​ Various elements of the visualization system (e.g., the structured light projector 706 and the camera 720 of the visualization system 100, 160, 500, 2108) can be used to generate visualization data of the dissected organ, thereby generating a virtual 3D construct 4130 of the dissected organ.

[0334] As described herein, structured light in the form of stripes or lines can be projected, for example, from a light source and / or projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contours of the surface 705. The camera 720 of the imaging device 120 ​ ) can be configured to detect the projected pattern of light on the surface 705, for example. The manner in which the projected pattern deforms upon striking the surface 705 allows the vision system to calculate depth and surface information of the target anatomical structure.

[0335] ​ A logic flow diagram of a process 4100 configured in accordance with the delineation control program or logic of at least one aspect of the present disclosure. In various instances, the process 4100 identifies 4101 a surgical procedure and identifies 4102 an anatomical organ targeted by the surgical procedure. The process 4100 further generates 4104 a virtual 3D construct 4130 of at least a portion of the anatomical organ, identifies 4105 an anatomical structure of the at least a portion of the anatomical organ relevant to the surgical procedure, couples 4106 the anatomical structure to the virtual 3D construct 4130, and superimposes 4107 a layout plan of the surgical procedure determined based on the anatomical structure onto the virtual 3D construct 4130.

[0336] One or more aspects of the process 4100 can be performed by one or more of the control circuits described in the present disclosure (e.g., the control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4100 are performed by the control circuit 400 of the control circuit 400 ​ , which includes a processor and a memory storing a set of computer executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4100. Additionally or alternatively, one or more aspects of the process 4100 can be performed by a combinational logic circuit (e.g., the control circuit 410 of the control circuit 410 ​ ) and / or a sequential logic circuit (e.g., the control circuit 420 of the control circuit 420 ​by the control circuitry of the visualization system (e.g., visualization system 100, 160, 500, 2108). In at least one example, the database is stored in a cloud-based system (e.g., cloud 2104 that can include remote servers 2113 coupled to storage 2105). In at least one example, the database includes a hospital EMR.

[0337] In various aspects, process 4100 can be implemented by a computer-implemented interactive surgical system 2100 (e.g., computer-implemented interactive surgical system 2100 of FIG. 21) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include remote servers 2113 coupled to storage 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with cloud 2104, which can include remote servers 2113. The control circuitry executing one or more aspects of process 4100 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108). ​

[0338] The control circuitry executing one or more aspects of process 4100 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can identify 4101 the surgical procedure and / or identify 4102 the anatomical organ targeted by the surgical procedure by retrieving such information from a database storing the information or directly from user input. In at least one example, the database is stored in a cloud-based system (e.g., cloud 2104 that can include remote servers 2113 coupled to storage 2105). In at least one example, the database includes a hospital EMR.

[0339] In one aspect, surgical system 2200 includes a surgical hub 2236 connected to a plurality of operating theater devices, such as a visualization system (e.g., visualization system 100, 160, 500, 2108) located in an operating room. In at least one example, surgical hub 2236 includes a communication interface for communicably coupling surgical hub 2236 to a visualization system, cloud 2204, and / or remote servers 2213. The control circuitry of surgical hub 2236 executing one or more aspects of process 4100 can identify 4101 the surgical procedure and / or identify 4102 the anatomical organ targeted by the surgical procedure by retrieving such information from a database stored in cloud 2204 and / or remote servers 2213.

[0340] The control circuitry executing one or more aspects of process 4100 can cause a visualization system (e.g., visualization system 100, 160, 500, 2108) to perform an initial scan of at least a portion of the anatomical organ to generate 4104 a three-dimensional (“3D”) construct 4130 of at least a portion of the anatomical organ targeted by the surgical procedure. In at least one example, the 3D construct 4130 is generated by a visualization system (e.g., visualization system 100, 160, 500, 2108) that is communicably coupled to a surgical hub 2236 (e.g., surgical hub 2236 of FIG. 22).​​ In the illustrated example, the anatomical organ is a stomach 4110. The control circuitry can cause one or more elements of the visualization system (e.g., the structured light projector 706 and the camera 720 utilizing structured light 4111) to generate visualization data by performing a scan of at least a portion of the anatomical organ as the camera is introduced into the body. The 3D construct of at least a portion of the anatomical organ can be generated utilizing the current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), visualization data from prior similar surgical procedures performed on the same or other patients, and / or user input.

[0341] Further, the control circuitry executing one or more aspects of the process 4100 identifies 4105 an anatomical structure of at least a portion of the anatomical organ relevant to the surgical procedure. In at least one example, a user can select the anatomical structure using any suitable input device. Additionally or alternatively, the visualization system can include one or more imaging devices 120 having spectral cameras (e.g., hyperspectral cameras, multispectral cameras, or selective spectral cameras) configured to detect reflected spectral waveforms and generate images based on molecular responses to different wavelengths. The control circuitry can utilize light absorption or refraction properties of tissue to distinguish different tissue types of the anatomical organ, thereby identifying the relevant anatomical structure. Further, the control circuitry can utilize the current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), stored visualization data from prior similar surgical procedures performed on the same or other patients, and / or user input to identify the relevant anatomical structure.

[0342] The identified anatomical structure can be an anatomical structure in the surgical field of view, and / or the anatomical structure is selected by a user. In various examples, the location tracking of the relevant anatomical structure can extend beyond the current visible view of the camera directed at the surgical field of view. In one example, this is achieved by using common visible coupling landmarks or by using secondary coupling motion tracking. For example, secondary tracking can be accomplished by secondary imaging sources, computation of range motion, and / or by pre-established beacons measured by a second visualization system.

[0343] As described above in connection with ​In further detail, the visualization system can utilize a structured light projector 706 to project an array of patterns or lines, where a camera 720 can determine a distance to a target location. The visualization system can then emit a pattern or line of known size at a set distance equal to the determined distance. Further, a spectral camera can determine a size of the pattern, which can vary depending on the light absorption or refraction properties of the tissue at the target location. A difference between the known size and the determined size is indicative of a tissue density at the target location, which is indicative of a tissue type at the target location. The control circuit executing one or more aspects of the process 4100 can identify a relevant anatomical structure based at least in part on the determined tissue density at the target location.

[0344] In at least one example, a detected tissue density anomaly can be associated with a disease state. Further, the control circuit selects, updates, or modifies one or more settings of a surgical instrument that treats tissue based on the tissue density detected via the visualization data. For example, the control circuit can change various clamp and / or firing parameters of a surgical stapler used to staple and cut tissue. In at least one example, the control circuit can slow a firing sequence and / or allow more clamp time based on the tissue density detected by the visualization data. In various examples, the control circuit can alert a user of the surgical instrument of an abnormal tissue density by, for example, displaying instructions on a screen to decrease bite size, increase or decrease energy delivery output of an electrosurgical instrument, and adjust the amount of jaw closure. In another example, if the visualization data indicates that the tissue is adipose tissue, the instructions can be to increase power to decrease energy application time.

[0345] Further, the identification of the type of surgical procedure can facilitate the identification of a target organ by the control circuit. For example, if the procedure is a left upper lobectomy, the lung is likely the target organ. Thus, the control circuit will only consider visualization data and non-visualization data related to the lung and / or tools typically used in such a procedure. Further, for example, knowledge of the type of procedure can better enable image fusion algorithms that inform tumor location and staple line placement.

[0346] In various aspects, knowledge of the surgical table position and / or insufflation pressure can be used by the control circuit executing one or more aspects of the process 4100 to establish a baseline position of a target anatomical organ and / or relevant anatomical structures identified from the visualization data. Movement of the surgical table (e.g., moving the patient from a flat position to an inverse Trendelenburg position) can result in deformation of the anatomical structures, which can be tracked and compared to the baseline to continuously inform the position and state of the target organ and / or relevant anatomical structures. Likewise, changes in insufflation pressure within a body cavity can disturb baseline visualization data of the target organ and / or relevant anatomical structures within the body cavity.

[0347] The control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more aspects of a process that derives baseline visualization data of a target organ and / or related anatomy of a patient on an operating table during a surgical procedure, determines a change in position of the operating table, and re-derives the baseline visualization data of the target organ and / or related anatomy of the patient in the new position.

[0348] Likewise, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more aspects of a process that derives baseline visualization data of a target organ and / or related anatomy of a patient on an operating table during a surgical procedure, determines a change in insufflation pressure in a patient’s body cavity, and re-derives the baseline visualization data of the target organ and / or related anatomy of the patient at the new insufflation pressure.

[0349] In various instances, the control circuit performing one or more aspects of process 4100 can couple the identified anatomical structures to the virtual 3D construct by superimposing landmarks or markers onto the virtual 3D construct to indicate the location of the anatomical structures, as shown in ​ The control circuit can also superimpose user-defined structures and tissue planes on the virtual 3D construct. In various aspects, a hierarchy of tissue types can be established to organize the anatomical structures identified on the virtual 3D construct. Table 1 provided below lists an exemplary hierarchy for the lungs and stomach.

[0350] ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​

[0351] In various aspects, the related anatomical structures identified on the virtual 3D construct can be renamed and / or repositioned by the user to correct errors or according to preference. In at least one example, the corrections can be voice activated. In at least one example, the corrections are recorded for future machine learning.

[0352] In addition to the above, the control circuit performing one or more aspects of process 4100 (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can superimpose 4107 a surgical procedure layout plan (e.g., layout plan 4120) onto the virtual 3D construct of the target organ (e.g., stomach 4110). In at least one example, the virtual 3D construct is displayed on a screen of the visualization system that is different from the screen that displays the real-time feed / view of the surgical field. In another example, one screen can alternately display the real-time feeds of the surgical field and the 3D construct. In such an example, the user can use any suitable input device to alternate between the two views.

[0353] In ​In the illustrated example, the control circuit has determined that the surgical procedure is a sleeve gastrectomy and that the target organ is the stomach. In the initial scan of the abdominal cavity, the control circuit uses the visualization data (e.g., structured light data and / or spectral data) to identify the stomach, liver, spleen, greater curvature of the stomach, and pylorus. This is informed by knowledge of the procedure and structures of interest.

[0354] The visualization data (e.g., structured light data and / or spectral data) can be utilized by the control circuit to identify the stomach 4110, liver, and / or spleen by comparing the current structured light data to stored structured light data previously associated with such organs. In at least one example, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can utilize structured light data representative of the characteristic anatomical profile of an organ and / or spectral data representative of characteristic subsurface tissue features to identify anatomical structures relevant to a surgical procedure layout plan (e.g., layout plan 4120).

[0355] In at least one example, the visualization data can be used to identify the hepatic portal vein 4112 indicative of the location 4113 of the pylorus 4131, identify the epiploic vessels 4114 indicative of the location 4115 of the greater curvature of the stomach 4110, identify the bend in the right gastric vein 4116 indicative of the location 4117 of the angle of Treitz 4132, and / or identify the location 4119 of the His angle 4121. The control circuit can assign a landmark to one or more of the identified locations. In at least one example, as ​ As illustrated, the control circuit causes the visualization system to overlay the landmarks on the locations 4113, 4117, 4119 on the virtual 3D construct of the stomach 4110 generated using the visualization data, as described above. In various aspects, the landmarks can be overlaid synchronously on the virtual 3D construct and the surgical field view to allow the user to toggle between the views without losing sight of the landmarks. The user can zoom out in the view of the screen displaying the virtual 3D construct to display the overall layout plan or can zoom in to display a portion similar to the surgical field view. The control circuit can continuously track and update the landmarks.

[0356] In a sleeve gastrectomy, the surgeon typically sutures the stomach tissue 4 cm or about 4 cm from the pylorus. Prior to suturing, an energy device is introduced into the abdominal cavity of the patient at the beginning of the sleeve gastrectomy to dissect the epiploic artery and omentum away from the greater curvature and 4 cm or about 4 cm from the pylorus. As described above, the control circuit having identified the location 4113 can automatically cause the end effector of the energy device to be overlaid 4 cm or about 4 cm from the location 4113. The overlay of the end effector of the energy device or any suitable landmark 4 cm or about 4 cm from the pylorus identifies the starting location of the sleeve gastrectomy.

[0357] As the surgeon dissects along the greater curve of the stomach, the control circuitry causes the landmark at location 4113 and / or the superimposed end effector of the energy device to be removed. As the surgeon approaches the spleen, a distance indicator is automatically superimposed on the virtual 3D construct view and / or the surgical field view. The control circuitry can cause the distance indicator to identify a distance of 2 cm from the spleen. For example, when the dissection path reaches or is about to reach 2 cm from the spleen, the control circuitry can cause the distance indicator to flash and / or change color. The distance indicator superimposition remains until the user reaches location 4119 at the His angle 4121.

[0358] Referring to ​ Once the surgical stapler is introduced into the abdominal cavity, the control circuitry can utilize the visualization data to identify the pylorus 4131, the angularis incisura 4132, the greater curve 4133 of the stomach 4110, the lesser curve 4134 of the stomach 4110, and / or other anatomical structures relevant to a sleeve gastrectomy. A superimposition of the bougie can also be displayed. Introduction of a surgical instrument into a body cavity, e.g., introduction of a surgical stapler into an abdominal cavity, can be detected by the control circuitry from the visualization data indicating a visual cue on the end effector, such as a unique color, marking, and / or shape. The control circuitry can identify the surgical instrument in a database storing such visual cues and corresponding visual cues. Alternatively, the control circuitry can prompt the user to identify the surgical instrument inserted into the body cavity. Alternatively, a surgical trocar facilitating access to the body cavity can include one or more sensors for detecting a surgical instrument inserted therethrough. In at least one example, the sensor includes an RFID reader configured to identify the surgical instrument from an RFID chip on the surgical instrument.

[0359] In addition to identifying landmarks of relevant anatomical structures, the visualization system can superimpose a surgical layout plan 4135 onto the 3D key structures and / or the surgical field view, which can be in the form of a suggested treatment path. In ​ In the example, the surgical procedure is a sleeve gastrectomy, and the surgical layout plan 4135 is in the form of three resection paths 4136, 4137, 4138 and the corresponding resulting volume of the sleeve.

[0360] As ​ indicated, distances (a, al, a2) from the pylorus 4131 to the starting points for forming a sleeve. Each starting point produces a different sleeve size (e.g., 400 cc, 425 cc, 450 cc for starting points 4146, 4147, 4148, respectively, at distances a, al, a2 from the pylorus 4131). In one example, the control circuitry prompts the user to input a size selection, and in response, presents a surgical layout plan, which can be in the form of a resection path, that produces the selected sleeve size. In another example, as ​As shown, the control circuit presents a plurality of resection paths 4136, 4137, 4138 and corresponding sleeve sizes. The user can then select one of the recommended resection paths 4136, 4137, 4138, and in response, the control circuit removes the non-selected resection paths.

[0361] In yet another example, the control circuit allows the user to make adjustments to the recommended resection path on a screen that shows the resection path superimposed on the virtual 3D construct and / or the surgical field of view. The control circuit can calculate the sleeve size based on these adjustments. Alternatively, in another example, the user is allowed to select a starting point to form the sleeve at a desired distance from the pylorus 4131. In response, the control circuit calculates the sleeve size based on the selected starting point.

[0362] For example, presenting the resection path can be accomplished by causing the visualization system to superimpose the resection path onto the virtual 3D construct view and / or the surgical field view. Conversely, removing the recommended resection path can be accomplished by causing the visualization system to remove the superimposition of such resection path from the virtual 3D construct view and / or the surgical field view.

[0363] Still referring to ​ In certain examples, once the end effector of the surgical stapler clamps stomach tissue selected from between the starting points 4146, 4147, 4147 selected from the recommended starting points and the end position 4140 at a predefined distance from the notch angle 4132, the control circuit presents information about clamping and / or firing the surgical stapler. In at least one example, as ​ As shown, a composite data set 4012 from the visualization data 4010 and the instrument data 4011 can be displayed. Additionally or alternatively, the FTC value and / or the FTF value can be displayed. For example, the current value of the FTC— represented by the circle 4020— can be depicted in real-time against a gauge 4021 with an indicator 4022 representing the best practice FTC. Likewise, the current value of the FTF— represented by the circle 4023— can be depicted against a gauge 4024 with an indicator 4025 representing the best practice FTF.

[0364] After the surgical stapler is fired, a recommendation for a new cartridge selection can be presented on the screen of the surgical stapler or any screen of the visualization system, as described in more detail below. As the surgical stapler is removed from the abdominal cavity, reloaded with the selected staple cartridge, and reintroduced into the abdominal cavity, a distance indicator— identifying a constant distance (d) from a plurality of points along the lesser curvature 4134 of the stomach 4110 to the selected resection path— is superimposed on the virtual 3D construct view and / or the surgical field view. To ensure proper orientation of the end effector of the surgical stapler, the distance from the target to the distal end of the end effector of the surgical stapler and the distance from the proximal end to the previously fired staple line is superimposed on the virtual 3D construct view and / or the surgical field view. This process is repeated until resection is complete.

[0365] One or more distances recommended and / or calculated by the control circuit can be determined based on stored data. In at least one example, the stored data includes preoperative data, user preference data, and / or data from previously performed surgical procedures by the user or other users.

[0366] Referring to ​ In accordance with at least one aspect of the disclosure, the process 4150 depicts a control program or logic configuration for recommending a resection path for removal of a portion of an anatomical organ. The process 4150 identifies 4151 an anatomical organ targeted by a surgical procedure, identifies 4152 an anatomical structure of the anatomical organ relevant to the surgical procedure, and recommends 4153 a surgical resection path for removal of a portion of the anatomical organ by a surgical instrument, as described in more detail elsewhere herein in connection with the process 4100. The surgical resection path is determined based on the anatomical structure. In at least one example, the surgical resection path includes different starting points.

[0367] One or more aspects of the process 4150 can be performed by one or more of the control circuits described in the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4150 are performed by a control circuit (e.g., ​ the control circuit 400) including a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4150. Additionally or alternatively, one or more aspects of the process 4150 can be performed by combinational logic circuitry (e.g., ​ the control circuit 410) and / or sequential logic circuitry (e.g., ​The process 4150 can be performed by the control circuit 420) executing. Further, the process 4150 can be performed by any suitable circuit having any suitable hardware and / or software components that can be located in or associated with various suitable systems described in the present disclosure.

[0368] Referring to ​ , the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of the process 4100 or the process 4150 can utilize dynamic visualization data to update or modify the surgical procedure layout plan in real-time during implementation. In at least one example, the control circuit modifies a set resection path (e.g., set resection path 4202) for removal of a portion of an organ or an abnormality (e.g., a tumor or a region) by a surgical instrument based on dynamic visualization data from one or more imaging devices of a visualization system (e.g., visualization system 100, 160, 500, 2108) that tracks the progress of resected tissue and surrounding tissue. The resection path modification can be triggered by the location of a critical structure (e.g., a blood vessel) moving into the resection path. For example, a tissue resection procedure can sometimes cause tissue inflammation that changes the shape and / or volume of tissue, which can cause a critical structure (e.g., a blood vessel) to shift. The dynamic visualization data enables the control circuit to detect changes in the location and / or volume of the critical structure and / or related anatomy in the vicinity of the set resection path. If the changes in location and / or volume cause the critical structure to move into the resection path or within a safety margin of the resection path, the control circuit modifies the set resection path by selecting or at least suggesting an alternative resection path for the surgical instrument. ​ ​

[0369] ​ A real-time view 4201 of a surgical field on a screen 4230 of a visualization system is shown. A surgical instrument 4200 is introduced into the surgical field to remove a target region 4203. An initial planned layout 4209 for removal of the region is superimposed on the real-time view 4201, as shown in a magnified view of the region 4203. The region 4203 is surrounded by critical structures 4205, 4206, 4207, 4208. As shown, the initial planned layout 4209 extends a resection path around the region 4203 with a predefined safety margin from the region 4203. The resection path avoids crossing or traversing the critical structures by extending on the outside (e.g., critical structure 4208) or inside (e.g., critical structure 4206) of the critical structures. As described above, the initial planned layout 4209 is determined by the control circuit based on visualization data from a visualization system. ​ ​

[0370] ​ ​​​​A real-time view 4201'of the surgical field on the screen 4230 of the visualization system at a later time (00:43) is shown. The end effector 4202 of the surgical instrument 4200 resects tissue along a predefined resection path defined by the layout plan 4209. For example, a volume change of tissue including region 4203 due to tissue inflammation causes key structures 4206 and 4208 to shift into the predefined resection path. In response, the control circuit recommends an alternative resection path 4210 that bypasses the key structures 4206, 4208, which protects the key structures 4206, 4208 from damage, as shown. ​ In various examples, an alternative resection path can be recommended to optimize the amount of healthy tissue to be preserved and provide guidance to the user to ensure they do not touch key structures, which will minimize bleeding and, therefore, reduce the surgical time and stress to handle unexpected situations while balancing the impact on the remaining organ volume.

[0371] In various aspects, the control circuit executing one or more aspects of one or more processes described by the present disclosure can receive and / or derive visualization data from a plurality of imaging devices of the visualization system. The visualization data facilitates tracking of key structures outside the real-time view of the surgical field. Common landmarks can allow the control circuit to merge the visualization data from the plurality of imaging devices of the visualization system. In at least one example, secondary tracking of key structures outside the real-time view of the surgical field can be achieved, for example, through a secondary imaging source, a calculation of range motion, or through a pre-established beacon / landmark measured by a second system.

[0372] Referring generally to ​ In accordance with at least one aspect of the present disclosure, the logic flow diagram of the process 4300 depicts a control procedure or logic configuration for presenting or overlaying parameters of a surgical instrument onto or near a recommended surgical resection path. The process 4300 is generally performed during a surgical procedure and includes identifying 4301 an anatomical organ targeted by the surgical procedure, identifying 4302 anatomical structures related to the surgical procedure from visualization data from at least one imaging device, and recommending 4303 a surgical resection path for removing a portion of the anatomical organ by a surgical instrument. In at least one example, the surgical resection path is determined based on the anatomical structures. The process 4300 further includes presenting 4304 parameters of the surgical instrument according to the surgical resection path. Additionally or alternatively, the process 4300 further includes adjusting 4305 parameters of the surgical instrument according to the surgical resection path.

[0373] One or more aspects of the process 4300 can be performed by one or more of the control circuits described by the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4300 are performed by a control circuit (e.g.,​ The control circuit 400 executes the process 4030, which includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4030. Alternatively or additionally, one or more aspects of the process 4300 may be executed by combinational logic circuitry (e.g., ​ The control circuit 410) and / or sequential logic circuit (e.g., ​ The process 4300 is executed by the control circuit 420. Furthermore, the process 4300 can be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0374] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) of one or more aspects of the execution process 4300 may identify 4301 the anatomical organ targeted by the surgery, identify 4302 the anatomical structures associated with the surgery based on visualization data from at least one imaging device of a visualization system (e.g., visualization systems 100, 160, 500, 2108), and / or recommend 4303 a surgical resection path for removing a portion of the anatomical organ by means of a surgical instrument (e.g., surgical instrument 4600), as elsewhere herein in conjunction with process 4150 ( ​ ), 4100 ​ As described above. Furthermore, the control circuitry of one or more aspects of the execution process 4300 may recommend or suggest one or more parameters of the surgical instrument based on the recommended surgical resection path 4303. In at least one example, the control circuitry presents suggested parameters 4304 of the surgical instrument by superimposing such parameters onto or near the recommended surgical path, such as... ​ and ​ As shown.

[0375] ​ A virtual 3D structure 4130 of the stomach of a patient undergoing sleeve gastrectomy using surgical instruments 4600 is shown according to at least one aspect of this disclosure. (As in conjunction with...) ​In further detail, various elements of the visualization system (e.g., visualization system 100, 160, 500, 2108) such as the structured light projector 706 and the camera 720 can be used to generate visualization data to form a virtual 3D construct 4130. Relevant anatomical structures (e.g., the pylorus 4131, the angularis incisura 4132, the His angle 4121) are identified from the visualization data from one or more imaging devices of the visualization system. In at least one example, landmarks are assigned to locations 4113, 4117, 4119 of such anatomical structures by superimposing the landmarks onto the virtual 3D construct 4130.

[0376] Further, based on the identified anatomical structures, a surgical resection path 4312 is recommended 4303. In at least one example, the control circuit superimposes the surgical resection path 4312 onto the virtual 3D construct 4130 as shown in ​ As described in further detail elsewhere herein, the recommended surgical path can be automatically adjusted based on a desired volume output. Further, the projected margin can be automatically adjusted based on critical structures and / or tissue abnormalities automatically identified by the control circuit from the visualization data.

[0377] In various aspects, the control circuit executing at least one aspect of the process 4300 presents a parameter 4314 of a surgical instrument selected according to the surgical resection path 4312 recommended 4303. In ​ In the example shown, the parameter 4314 indicates that a staple cartridge for use with the surgical instrument 4600 is automatically selected when performing a sleeve gastrectomy based on the surgical resection path recommended 4303. In at least one example, the parameter 4314 includes at least one of a staple cartridge size, a staple cartridge color, a staple cartridge type, and a staple cartridge length. In at least one example, the control circuit presents 4304 the parameter 4314 of the surgical instrument 4600 by superimposing such parameter onto or near the surgical resection path 4312 recommended 4303 as shown in ​ and ​ .

[0378] In various aspects, the control circuit executing at least one aspect of the process 4300 presents a tissue parameter 4315 along one or more portions of the surgical resection path 4312. In ​ In the example shown, the tissue parameter 4315 is a tissue thickness presented by displaying a cross-section taken along line A-A which represents a tissue thickness along at least a portion of the surgical resection path 4312. In various aspects, a staple cartridge utilized by the surgical instrument 4600 can be selected according to the tissue parameter 4315. For example, as shown in ​As shown, the black staple cartridge including the larger staple size is selected for the thicker antrum muscle tissue, while the green staple cartridge including the smaller staple size is selected for the cardiac muscle tissue of the body and fundus portions.

[0379] The tissue parameters 4315 include at least one of tissue thickness, tissue type, and volume results of the sleeve gastrectomy resulting from the recommended surgical resection path 4312. The tissue parameters 4315 can be derived from previously captured CT, ultrasound, and / or MRI images of the patient’s organ and / or from previously known average tissue thicknesses. In at least one example, the surgical instrument 4600 is a smart instrument (similar to the smart instrument 2112), and the tissue thickness and / or selected staple cartridge information is transmitted to the surgical instrument 4600 for optimization of closure settings, firing settings, and / or any other suitable surgical instrument settings. In one example, the tissue thickness and / or selected staple cartridge information can be transmitted from a surgical hub (e.g., surgical hub 2106, 2122) to the surgical instrument 4600, which is in communication with a visualization system (e.g., visualization system 100, 160, 500, 2108) and the surgical instrument 4600, as described in connection with FIGS. 21-22. In one example, the surgical hub 2106, 2122 can be in communication with the visualization system 100, 160, 500, 2108 and the surgical instrument 4600 via a wired or wireless connection. ​

[0380] In various examples, the control circuit executing at least one aspect of the process 4300 recommends an arrangement 4317 of two or more staple cartridge sizes (e.g., 45 mm and 60 mm) according to tissue thicknesses determined along at least a portion of the surgical resection path 4312. Further, as ​ As shown, the control circuit can further present the arrangement 4317 along the recommended 4303 surgical resection path 4312. Alternatively, the control circuit can present a suitable arrangement 4317 along a user-selected surgical resection path. As described above, the control circuit can determine tissue thicknesses along a user-selected resection path and recommend a staple cartridge arrangement according to the tissue thicknesses.

[0381] In various aspects, the control circuit executing one or more aspects of the process 4300 can recommend a surgical resection path, or optimize a selected surgical resection path, to minimize the number of staple cartridges in the arrangement 4317 without compromising the resulting sleeve gastrectomy size that exceeds a predetermined threshold. Reducing the number of expended cartridges reduces surgical time and cost, and reduces trauma to the patient.

[0382] Still referring to ​ ​Arrangement 4317 includes a first staple cartridge 4352 and a last staple cartridge 4353 defining the start and end points of surgical resection path 4312. If only a small portion of the last staple cartridge 4353 of the recommended staple cartridge arrangement 4317 is needed, control circuitry can adjust surgical resection path 4312 to eliminate the need for the last staple cartridge 4353 without compromising the size of the resulting sleeve stomach beyond a predetermined threshold.

[0383] In various examples, the control circuitry of at least one aspect of the execution process 4300 presents a virtual firing of the recommended stapling magazine arrangement 4317, which virtually divides the virtual 3D structure 4130 into a retention portion 4318 and a removal portion 4319, such as ​ As shown. The retained portion 4318 is a virtual representation of a sleeve stomach generated by implementing the recommended surgical resection path 4312 through the firing of the staple cartridge arrangement 4317. Control circuitry may further determine the estimated volumes of the retained portion 4318 and / or the removed portion 4319. The volume of the retained portion 4318 represents the volume of the resulting sleeve stomach. In at least one example, the volumes of the retained portion 4318 and / or the removed portion 4319 are derived from visualization data. In another example, the volumes of the retained portion 4318 and / or the removed portion 4319 are determined by a database storing the retained volume, the removed portion volume, and the corresponding surgical resection path. The database may be constructed from previous surgeries performed on organs of the same or at least similar size, which have been resected using the same or at least similar resection paths.

[0384] In various examples, a combination of pre-determined average tissue thickness data based on organ situational awareness, as described in more detail above, combined with volumetric analysis from visualization sources and secondary imaging from CT, MRI, and / or ultrasound, can be used to select the first staple cartridge for placement 4317, if available to the patient. In addition to visualization data, firing of subsequent staple cartridges in placement 4317 can be optimized using instrument data from previous firings. For example, instrument data that can be used to supplement volumetric measurements includes FTF, FTC, current consumption of the motor driving firing and / or closure, end effector closure gap, firing rate, tissue impedance measurements on the jaws, and / or waiting or pause times during surgical instrument use.

[0385] In various examples, visualization data (e.g., structured light data) can be used to track changes in surface geometry within tissue treated by a surgical instrument (e.g., surgical instrument 4600). Additionally, visualization data (e.g., spectral data) can be used to track key structures beneath the tissue surface. Structured and / or spectral data can be used to maintain established instrument-tissue contact throughout the tissue treatment.

[0386] In at least one example, the end effector 4642 of the surgical instrument 4600 can be used to grasp tissue between its jaws. Once the desired tissue-instrument contact is confirmed by user input, e.g., visualization data of the end effector and surrounding tissue associated with the desired tissue-instrument contact can be used to automatically maintain the desired tissue-surface contact in at least a portion of the tissue treatment. The desired tissue-surface contact can be automatically maintained by, e.g., slight manipulations to the position, orientation, and / or FTC parameters of the end effector 4642.

[0387] In the case where the surgical instrument 4600 is a hand-held surgical instrument, position and / or orientation manipulations can be provided to the user, e.g., in the form of instructions that can be presented on a display 4625 of the surgical instrument 4600 ​ ) when user manipulations are needed to reestablish the desired tissue-surface contact. Meanwhile, non-user manipulations, e.g., manipulations to FTC parameters and / or articulation angles, can be communicated from the surgical hub 2106 or visualization system 2108 to the controller 4620 of the surgical instrument 4600. The controller 4620 can then cause the motor driver 4626 to implement the desired manipulations. In the case where the surgical instrument 4600 is a surgical tool coupled to a robotic system 2110 of a robotic system 2110, e.g., position and / or orientation manipulations can be communicated from the surgical hub 2106 or visualization system 2108 to the robotic system 2110.

[0388] Referring primarily to ​ , firing of the surgical instrument 4600 loaded with the first cartridge 4652 of the staple cartridge arrangement 4317 is shown. In the first stage, as shown in ​ , the first landmark 4361 and the second landmark 4362 are superimposed on the surgical resection path 4312. The landmarks 4361, 4362 are spaced apart by a distance (dl) defined by the dimensions (e.g., 45) of the staple cartridge 4652, which represents the length of the staple line 4363 to be deployed onto the surgical resection path 4312 by the staple cartridge 4652. The control circuit executing one or more aspects of the process 4300 can employ visualization data, as described in greater detail elsewhere herein, to superimpose the landmarks 4361, 4362 onto the surgical resection path 4312 and continuously track and update their positions relative to the pre-determined critical structures (e.g., anatomical structures 4364, 4365, 4366, 4367).

[0389] During firing, as ​As shown, the staples of staple line 4363 are deployed into tissue, and the cutting member 4645 is advanced to cut tissue along the surgical resection path 4312 between the landmarks 4361, 4362. In various instances, the advancement of the cutting member 4645 causes the treated tissue to stretch and / or displace. Tissue stretch and / or displacement beyond a predetermined threshold indicates that the cutting member 4645 is moving too quickly through the treated tissue.

[0390] ​ is a logic flow diagram of a process 4170 that depicts a control program or logic configuration for adjusting a firing speed of a surgical instrument to address tissue stretch / displacement during firing. The process 4170 includes monitoring 4171 tissue stretch / displacement during firing of a surgical instrument, and if 4172 the tissue stretch / displacement is greater than or equal to a predetermined threshold, adjusting 4173 a firing parameter.

[0391] One or more aspects of the process 4170 can be performed by one or more of the control circuits described in the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4170 are performed by a control circuit (e.g., control circuit 400) that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4170. Additionally or alternatively, one or more aspects of the process 4170 can be performed by a combinational logic circuit (e.g., control circuit 410) and / or a sequential logic circuit (e.g., control circuit 420). ​ Additionally or alternatively, one or more aspects of the process 4170 can be performed by a combinational logic circuit (e.g., control circuit 410) and / or a sequential logic circuit (e.g., control circuit 420). ​ Additionally or alternatively, one or more aspects of the process 4170 can be performed by a combinational logic circuit (e.g., control circuit 410) and / or a sequential logic circuit (e.g., control circuit 420). ​ Additionally or alternatively, one or more aspects of the process 4170 can be performed by a combinational logic circuit (e.g., control circuit 410) and / or a sequential logic circuit (e.g., control circuit 420).

[0392] In various examples, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of the process 4170 monitors 4171 tissue stretch / displacement during firing of the surgical instrument 4600 using visualization data from a visualization system (e.g., visualization systems 100, 160, 500, 2108). In ​In the illustrated example, tissue stretch / displacement (d) is monitored 4171 by tracking distortion in the structured light grid projected onto the tissue during firing and / or tracking the position of the landmarks 4364, 4365, 4366, 4367 representing the location of the adjacent anatomy using the visualization data. Additionally or alternatively, tissue stretch (d) can be monitored 4171 by tracking the position of the landmark 4362 during firing. In ​ In the illustrated example, tissue stretch / displacement (d) is the difference between the distance (dl) between the landmarks 4361, 4362 during firing and the distance (d2) between the landmarks 4361, 4362 during firing. In any case, if 4172 the tissue stretch / displacement (d) is greater than or equal to a predetermined threshold, the control circuit adjusts 4173 the firing parameters of the surgical instrument 4600 to reduce the tissue stretch / displacement (d). For example, the control circuit can cause the controller 4620 to reduce the speed of the firing motor drive assembly 4604 by, for example, reducing the current draw of the firing motor 4602, for example, which reduces the rate of advancement of the cutting member 4645. Additionally or alternatively, the control circuit can cause the controller 4620 to pause the firing motor 4602 for a predetermined period of time to reduce the tissue stretch / displacement (d).

[0393] After firing, as ​ illustrated, the jaws of the end effector 4642 are released, and the stapled tissue is retracted due to the fired staples of the staple line 4363. ​ The projected staple line length defined by the distance (dl) and the actual staple line defined by the distance (d3) which is less than the distance (dl) are illustrated. The difference between the distances dl, d2 represents the retraction / displacement distance (d').

[0394] ​ is a logic flow diagram of a process 4180 depicting a control program or logic configuration for adjusting a recommended staple cartridge arrangement along a recommended surgical resection path. The process 4180 includes, after firing a staple cartridge of a recommended arrangement, monitoring 4081 retraction / displacement of the stapled tissue along the recommended surgical resection path, and adjusting a subsequent staple cartridge position of the recommended arrangement along the recommended surgical resection path.

[0395] One or more aspects of the process 4180 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by the present disclosure. In at least one example, one or more aspects of the process 4180 are performed by a control circuit (e.g., ​by a control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing process 4180. Process 4180 can be performed by any suitable circuit, such as a combinational logic circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) and / or a sequential logic circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620). ​ by a control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing process 4180. Process 4180 can be performed by any suitable circuit, such as a combinational logic circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) and / or a sequential logic circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620). ​ by a control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing process 4180. Process 4180 can be performed by any suitable circuit, such as a combinational logic circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) and / or a sequential logic circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620).

[0396] In various examples, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4180 monitors 4181 contraction / displacement of the stapled tissue along the recommended resection path 4312. In the illustrated example, the staple line 4363 is deployed from the staple cartridge of the arrangement 4317 into the tissue between the landmarks 4361, 4362. When the jaws of the end effector 4642 are loosened, the stapled tissue contracts / displaces a distance (d’). The distance (d’) is the difference between a distance (dl) representing the length of the staple line 4361 recommended by the arrangement 4317 and between the landmarks 4361, 4362 and a distance (d3) representing the actual length of the staple line 4363. ​

[0397] To avoid the gap between consecutive staple lines, the control circuit adjusts the subsequent staple cartridge position of the recommended arrangement 4317 along the recommended surgical resection path 4312. For example, as shown, the initially recommended staple line 4368 is removed and replaced with an updated staple line 4369 that extends through or covers the gap defined by the distance (d’). In various aspects, the contraction / displacement (d’) of the tissue is monitored 4181 after the jaws of the end effector 4642 are loosened by tracking the distortion in the structured light grid projected onto the tissue and / or tracking the landmarks 4364, 4365, 4366, 4367 representing the position of adjacent anatomical structures using the visualization data. Additionally or alternatively, the contraction distance (d’) of the tissue can be monitored 4181 by tracking the position of the landmark 4362. ​

[0398] ​​In various aspects, it can be desirable to corroborate visualization data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108) with non-visualization data from a non-visualization system, and vice versa. In one example, the non-visualization system can include a ventilator, which can be configured to measure non-visualization data of a patient’s lungs, such as volume, pressure, partial pressure of carbon dioxide (PCO2), partial pressure of oxygen (PO2), and the like. Corroborating the visualization data with the non-visualization data provides the clinician with greater confidence that the visualization data derived from the visualization system is accurate. Moreover, corroborating the visualization data with the non-visualization data allows the clinician to better identify post-operative complications, as well as determine the overall efficiency of the organ, as will be described in greater detail below. Corroborating can also aid in segmental resection or complex lobar resection without fissure.

[0399] In various aspects, a clinician can need to resect a portion of a patient’s organ to remove a critical structure, such as a tumor and / or other tissue. In one example, the patient’s organ can be a right lung. The clinician can need to resect a portion of the patient’s right lung to remove the unhealthy tissue. However, the clinician can not want to remove too much of the patient’s lung during the surgical procedure to ensure that the function of the lung is not too greatly impaired. The function of the lung can be assessed based on the peak lung volume per breath, which represents the peak lung volume. In determining how much of the lung can be safely removed, the clinician is limited by a pre-determined reduction in peak lung volume beyond which the lung will lose its viability, requiring a full organ resection.

[0400] In at least one example, the surface area and / or volume of the lung is estimated from visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). The lung surface area and / or volume can be estimated at the peak lung volume or peak lung volume per breath. In at least one example, the lung surface area and / or volume can be estimated at multiple points throughout the inhalation / exhalation cycle. In at least one aspect, prior to resecting a portion of the lung, the lung surface area and / or volume determined by the visualization system can be correlated with the lung volume determined by the ventilator using the visualization data and the non-visualization data. For example, the correlation data can be used to establish a mathematical relationship between the lung surface area and / or volume derived from the visualization data and the lung volume determined by the ventilator. This relationship can be used to estimate the size of the portion of the lung that can be removed while keeping the peak lung volume reduction at a value less than or equal to a pre-determined threshold that maintains the viability of the lung.

[0401] ​A logic flow diagram of a process 4750 for recommending surgical resection of a portion of an organ is shown in accordance with at least one aspect of the present disclosure. The process 4750 is generally performed during a surgical procedure. The process 4750 can include recommending 4752 a portion of an organ to be resected based on visualization data from a surgical visualization system, where resection of the portion is configured to produce an estimated volume reduction of the organ. The process 4750 can further include determining 4754 a first value of a non-visualized parameter of the organ prior to resection of the portion, and determining 4756 a second value of the non-visualized parameter of the organ after resection of the portion. Additionally, in certain examples, the process 4750 can further include validating 4758 the pre-determined volume reduction based on the first value of the non-visualized parameter and the second value of the non-visualized parameter.

[0402] One or more aspects of the process 4750 can be performed by one or more of the control circuits described in the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4750 are performed by a control circuit 400 of the type including a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4750. ​ Additionally or alternatively, one or more aspects of the process 4750 can be performed by a combinational logic circuit (e.g., control circuit 410 of the type including logic gates) and / or a sequential logic circuit (e.g., control circuit 420 of the type including flip-flops and / or registers). Moreover, one or more aspects of the process 4750 can be performed by any suitable circuit of the type having any suitable hardware and / or software components that can be located in or associated with various suitable systems described in the present disclosure. ​ Additionally or alternatively, one or more aspects of the process 4750 can be performed by a combinational logic circuit (e.g., control circuit 410 of the type including logic gates) and / or a sequential logic circuit (e.g., control circuit 420 of the type including flip-flops and / or registers). Moreover, one or more aspects of the process 4750 can be performed by any suitable circuit of the type having any suitable hardware and / or software components that can be located in or associated with various suitable systems described in the present disclosure. ​ Additionally or alternatively, one or more aspects of the process 4750 can be performed by a combinational logic circuit (e.g., control circuit 410 of the type including logic gates) and / or a sequential logic circuit (e.g., control circuit 420 of the type including flip-flops and / or registers). Moreover, one or more aspects of the process 4750 can be performed by any suitable circuit of the type having any suitable hardware and / or software components that can be located in or associated with various suitable systems described in the present disclosure.

[0403] In various aspects, the process 4750 can be implemented by a computer- implemented interactive surgical system 2100 (shown in FIG. 27) including one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which can include the remote server 2113. The control circuit executing one or more aspects of the process 4750 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108). ​

[0404] ​ ​A patient's lung 4780 is shown. In one embodiment, a clinician can utilize an imaging device 4782 to emit 4784 a pattern 4785 of light, such as stripes, grid lines, and / or dots, onto the surface of the patient's right lung 4786 to enable determination of the topography or panorama of the surface of the patient's right lung 4786. The imaging device can be similar in various respects to the imaging device 120 ​ As described elsewhere herein, the projected array of light can be used to determine the shape defined by the surface of the patient's right lung 4786, and / or the motion of the patient's right lung 4786 intraoperatively. In one embodiment, the imaging device 4782 can be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system, such as the surgical visualization system 100, can utilize surface mapping logic 136 of the control circuit 133 as described elsewhere herein to determine the topography or panorama of the surface of the patient's right lung 4786.

[0405] A clinician can provide a surgical system, such as the surgical system 2100, with the type of procedure to be performed, such as a right upper lobe lobectomy. In addition to providing the type of surgical procedure to be performed, the clinician can also provide the surgical system with a maximum desired volume of an organ to be removed during the surgical procedure. Based on the visualization data obtained from the imaging device 4782, the type of surgical procedure to be performed, and the maximum desired volume to be removed, the surgical system can recommend a resection path 4788 to remove a portion 4790 of the right lung that satisfies all of the clinician inputs. Other methods of recommending a surgical resection path are described elsewhere herein. The surgical system can consider any number of additional parameters in order to recommend the resection path 4788.

[0406] In various instances, it can be desirable to ensure that the volume of the resected organ produces the desired volume reduction of the patient's organ. In order to verify that the volume resected produces the desired volume reduction, non-visualization data from a non-visualization system can be utilized. In one embodiment, a ventilator can be used to measure the peak lung volume in a patient over time.

[0407] In at least one example, a clinician can utilize the surgical system 2100 to remove a lung tumor in a surgical procedure. As described above in connection with ​ the control circuit can identify the tumor from the visualization data and can recommend a surgical resection path that provides a safety margin around the tumor, as described above in connection with ​As described, the control circuitry can further estimate the lung volume at peak lung volume. A ventilator can be used to measure peak lung volume prior to surgery. Using a predetermined mathematical correlation between visually estimated lung volume and lung volume as detected by the ventilator, the control circuitry is able to estimate the reduction in peak lung volume associated with the removal of a portion of the lung, including a safe margin for the tumor and surrounding tissue. If the estimated reduction in lung volume exceeds a predetermined safe threshold, the control circuitry can alert clinicians and / or recommend different surgical resection pathways that result in a smaller reduction in lung volume.

[0408] FIG. 41C A graph 4800 shows the peak lung volume of a patient measured over time. Peak lung volume can be measured by the ventilator before this portion of the organ is removed (t1). FIG. 41C In the example described above, at time t1 prior to the resection of portion 4790, the peak lung volume was measured to be 6 L. In this case, where the surgical procedure to be performed is a right upper lobectomy, the clinician may wish to remove only the volume of the patient's lung that results in a predetermined volume reduction, so that the patient's breathing capacity is not impaired. In one implementation, for example, the clinician may wish to remove a portion that results in a reduction of up to approximately 17% in the patient's peak lung volume. Based on the surgical procedure and the desired volume reduction, the surgical system may recommend a surgical resection path 4788 that achieves the removal of the lung portion while maintaining a peak lung volume greater than or equal to 83% of the unresected peak lung volume.

[0409] Utilize FIG. 41C The ventilator data shown allows clinicians to monitor peak lung volume over time, for example, before (4802) and after (4804) the removal of portion 4790 of the lung. At time t2, portion 4790 of the lung is removed along the recommended resection path 4788. As a result, the peak lung volume measured by the ventilator decreases. Clinicians can use the ventilator data (peak lung volume before removal 4802 and peak lung volume after removal 4804) to confirm this and ensure that the removed lung volume results in the desired reduction in lung volume. FIG. 41C As shown, after resection, peak lung volume decreased to 5 L, representing a decrease of approximately 17%, which is roughly the same as the expected volume reduction. Using ventilator data, clinicians have greater confidence that the actual volume reduction is consistent with the expected volume reduction achieved through the recommended surgical resection path 4788. In other implementations, in cases where there are discrepancies between non-visualized and visualized data, such as a larger-than-expected decrease in peak lung volume (over-resection) or a smaller-than-expected decrease in peak lung volume (under-resection), clinicians can determine whether appropriate action should be taken.

[0410] Now for reference FIG. 41BThe right lung 4792 of the patient is shown after the resection of the portion 4790. After the resection of the portion 4790, the clinician can inadvertently cause an air leak 4794, which results in air leaking into the space between the lung 4794 and the chest wall, resulting in a pneumothorax 4796. Because of the air leak 4794, the peak lung volume of the patient per breath will steadily decrease over time as the right lung 4792 collapses. A dynamic surface area / volume analysis of the lung can be performed using visualization data derived from a visualization system (e.g., visualization system 100, 160, 500, 2108) to detect the air leak by visually tracking changes in lung volume. The volume and / or surface area of the lung can be visually tracked at one or more points during the inhalation / exhalation cycle to detect volume changes indicative of the air leak 4794. In one embodiment, as described above, the projected light array from the imaging device 4782 can be used to monitor the movement of the patient's right lung 4786 over time, such as a decrease in size. In another embodiment, the surgical visualization system can utilize surface mapping logic, such as surface mapping logic 136, to determine the topography or panorama of the surface of the patient's right lung 4786 and monitor changes in the topography or panorama over time.

[0411] In one aspect, the clinician can utilize a non-visualization system, such as a ventilator, to corroborate the decrease in volume detected by the visualization system. Referring again to FIG. 41C As described above, the peak lung volume of the patient can be measured before 4802 and after the portion of the lung is resected to corroborate that the expected decrease in volume is consistent with the actual decrease in lung volume. In the example described above, in the event that an air leak inadvertently occurs, the peak lung volume can steadily decrease 4806 over time. In one instance, at time t2immediately after the portion is resected, the clinician can record that the peak lung volume decreased from 6 L to 5 L, which is approximately consistent with the expected decrease in lung volume. After the portion is resected, the surgical visualization system can monitor the lung volume of the patient over time. If the surgical visualization system determines that there is a change in volume, the clinician can again measure the peak lung volume, for example, at time t3. At time t3, the clinician can record that the peak lung volume decreased from 5 L to 4 L, which corroborates the data determined from the visualization system that there can be an air leak in the right lung 4792.

[0412] Further, the control circuit can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, the organ efficiency can be determined by comparing the visualization data to the difference in non-visualization data before and after the portion is resected. In one example, the visualization system can generate a resection path to decrease the peak lung volume by 17%. The ventilator can be configured to measure the peak lung volume before the portion is resected and after the portion is resected. In this example, the ventilator can record that the peak lung volume decreased from 6 L to 5 L, which is approximately consistent with the expected decrease in lung volume. After the portion is resected, the surgical visualization system can monitor the lung volume of the patient over time. If the surgical visualization system determines that there is a change in volume, the clinician can again measure the peak lung volume, for example, at time t3. At time t3, the clinician can record that the peak lung volume decreased from 5 L to 4 L, which corroborates the data determined from the visualization system that there can be an air leak in the right lung 4792. FIG. 41CIn the illustrated case, there is a drop of about 17% in the peak lung volume (6L to 5L). Since the actual drop in lung volume (17%) is close to 1 : 1 with the expected drop in lung volume (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can generate a resection path to reduce the peak lung volume by 17%. However, as an example, the ventilator can measure a drop in peak lung volume greater than 17%, such as 25%. In this case, the clinician can determine that the lung is not functionally effective because resecting this portion of the lung results in a greater drop in peak lung volume than expected.

[0413] FIG. 40 A logic flow diagram of a process 4760 for estimating a reduction in volume of an organ due to removal of a selected portion of the organ is shown in accordance with at least one aspect of the present disclosure. Process 4760 is similar in many respects to process 4750. However, unlike process 4750, process 4760 relies on a clinician selecting or recommending a surgical resection path for removing a portion of an organ during a surgical procedure. Process 4760 includes receiving 4762 input from a user indicating a portion of an organ to be resected. Process 4760 further includes estimating 4764 a reduction in volume of the organ due to removal of the portion. In at least one example, the organ is a lung of a patient, and the estimated 4762 reduction in volume is a reduction in peak lung volume per breath of the patient’s lung. Visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108) can be employed to estimate the reduction in volume corresponding to the removal of the portion. Process 4760 can further include determining 4766 a first value of a non-visualized parameter of the organ prior to resecting the portion, and determining 4768 a second value of the non-visualized parameter of the organ after resecting the portion. Finally, process 4760 can further include validating 4768 the estimated amount of reduction in volume of the organ based on the first value of the non-visualized parameter and the second value of the non-visualized parameter.

[0414] One or more aspects of process 4760 can be performed by one or more of the control circuits described by the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4760 are performed by a control circuit (e.g., control circuit 400) that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4760. Additionally or alternatively, one or more aspects of process 4760 can be performed by combinational logic circuitry (e.g., control circuit 410) and / or sequential logic circuitry (e.g., control circuit 420). FIG. 2A FIG. 2B FIG. 2C ​​The control circuit 420) executes the process. Furthermore, one or more aspects of the process 4760 may be executed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with the various suitable systems described in this disclosure.

[0415] In all respects, process 4760 can be implemented via a computer-based interactive surgical system 2100. FIG. 19 The computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 communicating with the cloud 2104, which may include the remote server 2113. Control circuitry for one or more aspects of the execution process 4760 may be components of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0416] In one scenario, a clinician may provide input to a surgical visualization system (such as surgical visualization system 2100) indicating the portion of an organ to be removed. In another scenario, the clinician may draw a resection path on a virtual 3D structure of the organ (such as the virtual 3D structure generated during procedure 4104). In other scenarios, the visualization system may overlay a surgical layout plan, described in more detail elsewhere herein, which may be in the form of a proposed treatment path. The proposed treatment path may be based on the type of surgery being performed. In one embodiment, the proposed treatment path may recommend different starting points and different resection paths that the clinician can choose from, similar to resection paths 4146, 4147, 4148 described elsewhere herein. The recommended resection path may be determined by the visualization system to avoid certain critical structures such as arteries. The clinician may select a recommended resection path until the desired resection path for removing the portion of the organ is completed.

[0417] In one case, the surgical visualization system can determine an estimated volume reduction of the organ based on the selected resection path. After a predetermined portion is resected along the resection path, the clinician can wish to use non-visualization data to confirm that the actual reduction in volume corresponds to the estimated volume reduction based on the visualization data. In one embodiment, this confirmation can be accomplished using a similar procedure as described above with respect to process 4750 where the organ is a lung. The peak lung volume is measured before 4802 and after 4804 the lung resection, and the change in peak lung volume is compared to determine the actual drop in peak lung volume. In one example, the surgical visualization system can estimate a 17% reduction in peak lung volume based on the clinician's recommended resection path. Prior to resection, the clinician can record a peak lung volume of 6L (time ti). After resection, the clinician can record a peak lung volume of 5L (time t2), which is approximately a 17% drop in peak lung volume. Using this non-visualization / ventilator data, the clinician has greater confidence that the actual reduction in volume is consistent with the estimated volume reduction. In other cases, where there is a discrepancy between the non-visualization data and the visualization data, such as the peak lung volume dropping greater than expected (too much lung resection) or the peak lung volume dropping less than expected (not enough lung resection), the clinician can determine whether appropriate action should be taken.

[0418] Further, the control circuit can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, organ efficiency can be determined by comparing the visualization data to the difference in non-visualization data before and after resection of the portion. In one example, the visualization system can estimate a 17% reduction in peak lung volume based on the clinician's desired resection path. The ventilator can be configured to measure the peak lung volume before resection of the portion and after resection of the portion. In the case shown, there is approximately a 17% drop in peak lung volume (6L to 5L). Since the actual drop in lung volume (17%) is close to 1 : 1 with the estimated drop in lung volume (17%), the clinician can determine that the lung is functionally efficient. In another example, the visualization system can estimate a 17% reduction in peak lung volume based on the clinician's desired resection path. However, as an example, the ventilator can measure a drop in peak lung volume greater than 17%, such as 25%. In this case, the clinician can determine that the lung is not functionally efficient since resection of the portion of the lung resulted in a greater drop in peak lung volume than expected. FIG. 41C

[0419] As described above with respect to processes 4750, 4760, the clinician can use non-visualization data to confirm the visualization data, for example, by using a ventilator to measure the peak lung volume before and after resection of a portion of the lung. Another example of using non-visualization data to confirm the visualization data is through a capnogram. ​

[0420] FIG. 42 A plot 4810 of the partial pressure of carbon dioxide (PCO2) exhaled by the patient over time is shown. In other cases, the partial pressure of oxygen (PO2) exhaled by the patient can be measured over time. The plot 4810 shows the PCO2 measured prior to resection 4812, immediately after resection 4814, and one minute after resection 4816. In the example described above, where the surgical procedure to be performed is a right upper lobe resection, the visualization system can expect or estimate a 17% reduction in lung volume. The PCO2 level measured by the ventilator can be used to confirm this expected or estimated volume reduction. FIG. 42 In the example described above, where the surgical procedure to be performed is a right upper lobe resection, the visualization system can expect or estimate a 17% reduction in lung volume. The PCO2 level measured by the ventilator can be used to confirm this expected or estimated volume reduction.

[0421] With ventilator data as shown in FIG. 42 , the clinician can monitor the PCO2 of the patient over time, for example, prior to resection 4812 of the portion 4790 of the lung and immediately after resection 4814. At time t2, the portion 4790 of the lung has been resected, and thus, the PCO2 measured by the ventilator can decrease 4818. The clinician can use the ventilator data (pre-resection 4812 PCO2 (at tl) and post-resection 4814 PCO2 (at t2)) to confirm that the actual reduction in lung volume is consistent with the estimated or expected reduction in lung volume. As shown in FIG. 42 , immediately after resection 4814 of the portion 4790, the PCO2 decreases 4812, which can be measured as a decrease of about 17% in PCO2 (about 33.2 mmHg). Using this non-visualization / ventilator data, the clinician has greater confidence that the actual reduction in lung volume is consistent with the expected or estimated reduction in lung volume.

[0422] In other cases, the clinician can utilize the non-visualization / PCO2 data to determine differences when compared to the visualization data. In one case, immediately after resection 4814 of the portion 4790, at time t2, the PCO2 can be measured at 4820, which is higher than the PCO2 measured prior to resection 4812. The increase in PCO2 can be a result of an unintentional blockage of a bronchus during the surgical procedure, resulting in the accumulation of CO2 in the patient. In another case, immediately after resection 4814 of the portion 4790, at time t2, the PCO2 can be measured at 4822, which is lower than the PCO2 measured prior to resection 4812 and lower than expected. The decrease in PCO2 can be a result of an unintentional blockage of a blood vessel during the surgical procedure, resulting in less O2 being delivered to the body, and thus, less CO2 being produced. In either case, the clinician can take appropriate measures to remedy the situation.

[0423] Changes in PC02 can also be measured at times other than immediately after resection 4814, such as one minute after resection 4816 (e.g., at time t3). At time t3, other body functions, such as the kidneys, compensate for changes in PC02 due to resection. In this case, PC02 can be measured as about 40 mmHg, or approximately the same as the measurement before resection 4812. The difference between the measured and pre-resection 4812 PC02 at time t3 can indicate the unintentional obstruction discussed above. For example, at time t3, PC02 can be measured 4824 as higher than pre-resection 4812, indicating a likely unintentionally obstructed bronchus, or PC02 can be measured 4826 as lower than pre-resection 4812, indicating a likely unintentionally obstructed blood vessel.

[0424] Further, the control circuit can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, organ efficiency can be determined by comparing the visualization data to the difference in non-visualization data before and after resection of the portion. In one example, the visualization system can estimate a 17% reduction in lung capacity based on the clinician's desired resection path. The ventilator can be configured to measure PC02 before resection of the portion and after resection of the portion. The difference between the pre-resection and post-resection PC02 measurements can be compared to the estimated reduction in lung capacity to determine organ efficiency. FIG. 42 In the illustrated embodiment, PC02 drops by about 17% immediately after resection 4814. Since the PC02 drop (17%) is close to 1 : 1 with the estimated lung capacity drop (17%), the clinician can determine that the lung is functionally efficient. In another example, the visualization system can estimate a 17% reduction in lung capacity based on the clinician's desired resection path. However, as an example, the ventilator can measure a PC02 drop greater than 17%, such as 25%. In this case, the clinician can determine that the lung is not functionally efficient since resection of the portion of the lung resulted in a greater PC02 drop than expected.

[0425] In addition to the lung capacity peak and PC02 measurements described above, other non-visualization parameters can also be utilized, including blood pressure or EKG data. EKG data will provide approximate frequency data regarding arterial deformation. This frequency data with surface geometry changes in similar frequency ranges can help identify critical vascular structures.

[0426] As described above, it can be desirable to utilize non-visual data from a non-visualization system to corroborate visual data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). In the example described above, the non-visual data provided a means for corroborating the visual data after a portion of an organ had been resected. In some cases, it can be desirable to supplement the visual data with non-visual data prior to resecting a portion of an organ. In one example, the non-visual data can be used with the visual data to help determine a characteristic of an organ to be operated on. In one aspect, the characteristic can be an abnormality of the tissue of the organ that can not be suitable for cutting. The non-visual data and the visual data can help inform the surgical visualization system and the clinician of areas to avoid when planning a resection path of the organ. This can facilitate a segmental resection or a complex lobar resection without fissure.

[0427] FIG. 43 A logic flow diagram of a process 4850 for detecting a tissue abnormality based on visual data and non-visual data is shown in accordance with at least one aspect of the present disclosure. The process 4850 is generally performed during a surgical procedure. The process 4850 can include receiving 4852 first visual data from a surgical visualization system in a first state of an organ, and determining 4854 a first value of a non-visual parameter of the organ in the first state. The process 4850 can further include receiving 4856 second visual data from the surgical visualization system in a second state of the organ, and determining 4858 a second value of the non-visual parameter of the organ in the second state. The process can also include detecting 4860 a tissue abnormality based on the first visual data, the second visual data, the first value of the non-visual parameter, and the second value of the non-visual parameter.

[0428] One or more aspects of the process 4850 can be performed by one or more of the control circuits described in the present disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of the process 4850 are performed by a control circuit (e.g., control circuit 400) that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4850. Additionally or alternatively, one or more aspects of the process 4850 can be performed by combinational logic circuitry (e.g., control circuit 410) and / or sequential logic circuitry (e.g., control circuit 420). FIG. 2A FIG. 2B FIG. 2C ​​The process 4850 can be implemented by the control circuit 420 of the system 2100, for example. In addition, one or more aspects of the process 4850 can be performed by any suitable circuit having any suitable hardware and / or software components that can be located in or associated with various suitable systems described in the present disclosure.

[0429] In various aspects, the process 4850 can be implemented by a computer- implemented interactive surgical system 2100 including one or more surgical systems 2102 and a cloud-based system (e.g., the cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which can include the remote server 2113. The control circuitry performing one or more aspects of the process 4850 can be a component of a visualization system (e.g., the visualization systems 100, 160, 500, 2108). FIG. 19

[0430] FIG. 44A A right lung 4870 of a patient in a first state 4862 is shown. In one example, the first state 4862 can be a contracted state. In another example, the first state 4862 can be a collapsed state. An imaging device 4872 is shown inserted through a cavity 4874 in a chest wall 4876 of the patient. The clinician can utilize the imaging device 4872 to emit 4880 a pattern 4882 of light onto a surface of the right lung 4870, such as stripes, grid lines, and / or dots, to enable a determination of a topography or panorama of the surface of the right lung 4870 of the patient. The imaging device can be similar in various aspects to the imaging device 120 FIG. 1 As described elsewhere herein, the array of projected light is used to determine a shape defined by the surface of the right lung 4870 of the patient, and / or motion of the right lung 4870 of the patient intraoperatively. In one embodiment, the imaging device 4782 can be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system such as the surgical visualization system 100 can utilize surface mapping logic 136 of the control circuit 133 as described elsewhere herein to determine a topography or panorama of the surface of the right lung 4786 of the patient. At the first state 4862 of the right lung 4870, a ventilator can be used to measure a parameter of the right lung 4870, such as a first state pressure (PI, or positive end-expiratory pressure (PEEP)) or a first state volume (VI).

[0431] FIG. 44B ​The right lung 4870 of the patient in the second state 4864 is shown. In one example, the second state 4864 can be a partially inflated state. In another example, the second state 4864 can be a fully inflated state. The imaging device 4872 can be configured to continue to emit 4880 a pattern of light 4882 onto the surface of the lung 4870, enabling a determination of the topography or panorama of the surface of the right lung 4870 of the patient in the second state 4864. At the second state 4864 of the right lung 4870, the ventilator can be used to measure parameters of the right lung 4870, such as a second state pressure (P2) that is greater than the first state pressure PI and a second state volume (V2) that is greater than the first state volume VI.

[0432] Based on the surface topography determined from the surgical visualization system and the imaging device 4872, and the non-visualized data determined from the ventilator (pressure / volume), the surgical visualization system can be configured to determine tissue abnormalities of the right lung 4870. In one example, in the first state 4862, the imaging device 4872 can determine the first state 4662 topography of the right lung 4870 (shown in FIG. 44A and shown in more detail in FIG. 44C ), and the ventilator can determine the first state pressure / volume. In the second state 4864, the imaging device 4872 can determine the second state 4864 topography of the right lung 4870 (shown in FIG. 44B and shown in more detail in FIG. 44D ), and the ventilator can determine the second state pressure / volume that is greater than the first state pressure / volume due to the lung being partially or fully inflated. Based on the known pressure / volume increase, the visualization system can be configured to monitor the topography changes of the right lung 4870 according to the known pressure / volume increase. In one aspect, this pressure / volume measurement from the ventilator can be correlated to the surface deformation of the right lung 4870 to identify areas of disease within the lung to help inform stapler placement.

[0433] In one aspect, reference is made to FIG. 44B and FIG. 44D where the pressure increases from PI to P2 (and the volume increases from VI to V2), the surface topography determined from the structured light 4880 has changed compared to the first state 4862. In one example, the pattern of light 4882 can be dots, and as the lung size increases, the dots have spaced apart a distance from one another. In another example, the pattern of light 4882 can be grid lines, and as the lung size increases, the grid lines are spaced apart or contoured. Based on the known pressure increase, the imaging device can determine areas 4886 that have not changed according to the known pressure and volume increase. For example, in the imaging device 4872 emits the pattern of grid lines and dots 4882 onto the surface of the right lung 4870 (as shown in FIGS. 44A-44DIn the case of a known pressure / volume increase (as shown), the visualization system can be configured to monitor the profile of the grid lines and the positioning of the dots relative to each other. In the event that the visualization system notices that the spacing of the dots or the positioning and curvature of the grid lines is irregular, the visualization system can determine that these areas correspond to potential abnormalities of the tissue, such as areas or subsurface voids 4886 in which critical structures 4884, such as tumors, can be located. In one embodiment, with reference to the process 4100, where the process 4100 identifies 4105 an anatomical structure of at least a portion of an anatomical organ relevant to a surgical procedure, the process 4100 can identify abnormalities as described above and superimpose these abnormalities onto the 3D construct.

[0434] In one example, a patient can have emphysema, a lung disease that causes shortness of breath. In people with emphysema, the air sacs in the lungs (alveoli) are damaged, and over time, the inner walls of the air sacs weaken and break, creating larger air spaces instead of many small ones. This reduces the internal surface area of the lungs for O2 / CO2 exchange, thus reducing the amount of oxygen reaching the bloodstream. In addition, the damaged alveoli are dysfunctional, old air is trapped, and there is no room for fresh, oxygen-rich air to enter. The voids within the lungs of an emphysema patient represent areas with less tissue thickness, and thus will affect the stapling results in that area. The tissue is also weakened, which causes the alveoli to rupture, and is less able to hold staples through them.

[0435] As the lung with emphysema inflates and deflates, the areas with subsurface voids will have a different amount of deformation compared to healthy tissue as a function of pressure. With the process 4850 described above, these areas of weak tissue with subsurface voids can be detected to inform the clinician that they should avoid stapling through these areas, which can reduce the likelihood of postoperative air leaks. The tissue deformation capabilities of this process 4850 will allow the detection of these differences, allowing the surgeon to be guided when placing the stapler.

[0436] In a second example, a patient can have cancer. Prior to surgery, the tumor can have been irradiated, which damages the tissue as well as the surrounding tissue. Irradiation changes the properties of the tissue, typically making it harder and less compressible. If the surgeon needs to staple over this tissue, the change in tissue hardness should be taken into account when selecting the staple reload type (e.g., harder tissue will require a higher-formed staple).

[0437] As the lung inflates and deflates, the areas with harder tissue will have a different amount of deformation compared to healthy tissue, as the lung is less compliant in these areas. The tissue deformation capabilities of this process 4850 will allow the detection of these differences, allowing the surgeon to be guided when placing the stapler as well as selecting the cartridge / reload color.

[0438] In another aspect, a memory, such as memory 134, can be configured to store a surface topography of a lung at known pressures and volumes. In this case, an imaging device, such as imaging device 4872, can emit a pattern of light to determine the topography of the surface of the patient's lung at a first known pressure or volume. A surgical system, such as surgical system 2100, can be configured to compare the first determined topography at the known first pressure or volume to the topography stored in memory 134 at a given first pressure or volume. Based on this comparison, the visualization system can be configured to indicate potential abnormalities of the tissue only in a single state. The visualization system can record these potential abnormal areas and proceed to determine the topography of the surface of the patient's lung at a second known pressure or volume. The visualization system can compare the second determined surface topography to the topography stored in memory at a second given pressure or volume and the determined topography at the first known pressure or volume. If the visualization system determines a potential abnormal area that overlaps with the first determined potential abnormal area, the visualization system can be configured to indicate the overlapping area as a potential abnormality with greater confidence based on the comparison at the first and second known pressures or volumes.

[0439] In addition to the above, PO2 measurements from a ventilator can be compared to inflated lung volumes, such as V2, and deflated lung volumes, such as VI. Volume comparisons can utilize EKG data to compare inhalation and exhalation, which can be compared to blood oxygenation. This can also be compared to narcotic gas exchange measurements to determine the relationship of respiratory volume, oxygen uptake, and sedation. Additionally, EKG data can provide approximate frequency data regarding arterial deformation. This frequency data with surface geometry changes in a similar frequency range can help identify critical vascular structures.

[0440] In another embodiment, current tracking / surgical information can be compared to preoperative planning simulations. In challenging or high-risk surgeries, a clinician can utilize a preoperative patient scan to simulate a surgical approach. This data set can be compared in a display, such as display 146, to real-time measurements to help the surgeon be able to follow a specific preoperative plan based on training runs. This would require the ability to match fiducial landmarks between the preoperative scan / simulation and the current visualization. One approach can simply use boundary tracking of the subject. Insights into tissue type discrimination, relative tissue deformation assessment, or subsurface structure differences comparing how the current device-tissue interaction compares to previous interactions (per patient) or expected interactions (database or past patients) can be stored in a memory, such as memory 134.

[0441] In one embodiment, the surface geometry can be a function of tool position. The surface reference can be selected when no change in surface geometry is measured with each change in tool position. When the tool interacts with the tissue and deforms the surface geometry, the surgical system can calculate the change in surface geometry as a function of tool position. For a given change in tool position while in contact with the tissue, the change in tissue geometry can be different in areas with subsurface structures, such as critical structures 4884, than in areas without such structures, such as subsurface voids 4886. In one example, such as a thoracic procedure, this can be over airways but not just in parenchyma. The surgical system can calculate the running average of tool position change versus surface geometry change for a given patient using the surgical visualization system, giving a patient-specific difference, or can compare this value to a second set of previously collected data.

[0442] Exemplary Clinical Applications

[0443] The various surgical visualization systems disclosed herein can be used in one or more of the following clinical applications. The following clinical applications are non-exhaustive and merely illustrative applications for one or more of the various surgical visualization systems disclosed herein.

[0444] The surgical visualization systems as disclosed herein can be used in a variety of different types of procedures for different medical specialties, such as urology, gynecology, oncology, colorectal, thoracic, obesity / gastro, and hepatopancreaticobiliary (HPB), for example. For example, in urological procedures, such as prostatectomy, ureters can be detected in fat, for example, or connective tissue and / or nerves can be detected in fat. For example, in gynecological oncology procedures, such as hysterectomy, for example, and in colorectal procedures, such as low anterior resection (LAR) procedures, for example, ureters can be detected in fat and / or connective tissue. For example, in thoracic procedures, such as lobectomy, blood vessels can be detected in lung or connective tissue, and / or nerves can be detected in connective tissue (e.g., esophagostomy). In obesity procedures, blood vessels can be detected in fat. For example, in HPB procedures, such as hepatectomy or pancreatectomy, blood vessels can be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts can be detected in thin-walled (liver or pancreas) tissue.

[0445] In one example, a clinician can want to remove an endometrial myoma. From a preoperative magnetic resonance imaging (MRI) scan, the clinician can know that the endometrial myoma is located on the surface of the bowel. Thus, the clinician can want to know which tissues make up part of the bowel and which tissues make up part of the rectum during the procedure. In such a case, a surgical visualization system as disclosed herein can indicate different types of tissue (bowel vs. rectum) and communicate that information to the clinician via an imaging system. Further, the imaging system can determine a proximity of a surgical device to a selected tissue and communicate that proximity. In such a case, the surgical visualization system can provide increased surgical efficiency without serious complications.

[0446] In another example, a clinician (e.g., a gynecologist) can keep away from certain anatomical areas to avoid getting too close to critical structures, and thus the clinician can not be able to remove, for example, all endometriosis. A surgical visualization system as disclosed herein can enable the gynecologist to reduce the risk of getting too close to critical structures so that the gynecologist can get close enough with a surgical device to remove all endometriosis, which can improve patient outcomes (democratize surgical procedures). Such a system can enable a surgeon to "keep moving" during a surgical procedure rather than repeatedly stopping and restarting in order to identify areas to avoid, for example, particularly during the application of therapeutic energy such as ultrasonic or electrosurgical energy. In gynecological applications, the uterine artery and ureter are important critical structures, and the system can be particularly useful for hysterectomy and endometriosis surgery, taking into account the presentation and / or thickness of the tissues involved.

[0447] In another example, a clinician can have a risk of dissecting a blood vessel at too close a location, and thus this can impact the blood supply to lobes other than the target lobe. Further, anatomical differences from patient to patient can cause the anatomy to impact blood vessels (e.g., branches) of different lobes based on a particular patient. A surgical visualization system as disclosed herein can enable identification of the correct blood vessel at the desired location, which enables the clinician to dissect with appropriate anatomical certainty. For example, the system can confirm that the correct blood vessel is at the correct location, and then the clinician can safely separate the blood vessel.

[0448] In another example, a clinician can make multiple dissections before dissecting at the optimal location due to uncertainty in the anatomical structure of the blood vessel. However, it is desirable to dissect at the optimal location in the first instance because more dissections can increase the risk of bleeding. A surgical visualization system as disclosed herein can minimize the number of dissections by indicating the correct blood vessel and the optimal location for dissection. For example, the ureter and the main ligament are dense and present unique challenges during dissection. In such a case, it can be particularly desirable to minimize the number of dissections.

[0449] In another example, a clinician (e.g., a surgical oncologist) can want to know the identification of critical structures, the localization of cancer, the staging of cancer, and / or the assessment of tissue health. Such information is beyond what the clinician sees with the “naked eye.” A surgical visualization system as disclosed herein can determine such information intraoperatively and / or communicate such information to the clinician to strengthen intraoperative decision making and improve surgical outcomes. In certain instances, the surgical visualization system can be compatible with minimally invasive surgical (MIS) procedures using, for example, endoscopes or exoscopes, open surgical procedures, and / or robotic approaches.

[0450] In another example, a clinician (e.g., a surgical oncologist) can want to turn off one or more alerts regarding the proximity of a surgical tool to one or more critical structures to avoid being overly conservative during a surgical procedure. In other instances, the clinician can want to receive certain types of alerts such as haptic feedback (e.g., vibration / buzz) to indicate proximity and / or “no-fly zones” to stay sufficiently away from one or more critical structures. For example, a surgical visualization system as disclosed herein can provide flexibility based on the experience of the clinician and / or the desired aggressiveness of the procedure. In such instances, the system provides a balance between “knowing too much” and “knowing enough” to anticipate and avoid critical structures. The surgical visualization system can help plan the next steps during a surgical procedure.

[0451] Various aspects of the subject matter described herein are set out in the following numbered clauses.

[0452] Clause 1. A surgical system for a surgical procedure, the su...

Claims

1. A surgical system for use in surgical procedures, the surgical system comprising: At least one imaging device; and Control circuit, the control circuit being configured to: Identify the anatomical organ targeted by the surgical procedure; A virtual three-dimensional structure of at least a portion of the anatomical organ is generated based on visualization data generated by at least one imaging device during surgery, wherein the visualization data includes structured light data and spectral data, and wherein the structured light data is generated by at least one imaging device based on sensing a light pattern projected onto the surface of the anatomical organ. Identifying anatomical structures related to the surgery based on the visualization data from the at least one imaging device, wherein identifying anatomical structures includes identifying the anatomical contours of the anatomical organs based on the structured light data and identifying subsurface tissue features of the anatomical structures based on the spectral data; Couple the anatomical structure to the virtual three-dimensional construct; and The surgical layout plan, determined based on the anatomical structure, is superimposed on the virtual three-dimensional structure.

2. The surgical system according to claim 1, wherein, The layout plan includes the starting position of the cutting component of the end effector determined based on the anatomical structure.

3. The surgical system according to claim 1, wherein, The layout plan includes a surgical resection path for removing a portion of the anatomical organ, and wherein the surgical resection path is determined based on the anatomical structure.

4. The surgical system according to claim 3, wherein, The control circuit is configured to predict the volume of the anatomical organ based on the surgical resection path.

5. The surgical system according to claim 1, wherein, Identifying the anatomical structure of the anatomical organ includes distinguishing the tissue type of the anatomical organ.

6. The surgical system according to claim 1, wherein, Identify at least one of the anatomical organs and anatomical structures based on the stored data.

7. The surgical system according to claim 6, wherein, The stored data includes at least one of preoperative data, user preference data, and data from previously performed surgical procedures.

8. A surgical system for use with surgical instruments in surgical procedures, said surgical system comprising: At least one imaging device; and Control circuit, the control circuit being configured to: Identify the anatomical organ targeted by the surgical procedure; Identifying anatomical structures related to the surgical procedure based on visualization data from the at least one imaging device, wherein the visualization data includes structured light data and spectral data, wherein the structured light data is generated by the at least one imaging device based on sensing light patterns projected onto the surface of the anatomical organ, and wherein the anatomical structures are identified based on the anatomical contours of the anatomical organ contained in the visualization data and subsurface tissue features of the anatomical structures contained in the visualization data; and A recommended surgical resection path is provided for removing a portion of the anatomical organ using the surgical instruments, wherein the surgical resection path is determined based on the anatomical structure.

9. The surgical system according to claim 8, wherein, The surgical resection path includes different starting points.

10. The surgical system according to claim 8, wherein, The surgical resection path results in different organ volumes.

11. The surgical system according to claim 8, wherein, The surgical resection path is superimposed on a virtual three-dimensional structure of at least a portion of the anatomical organ.

12. The surgical system according to claim 8, wherein, Identifying the anatomical structure of the anatomical organ includes distinguishing the tissue type of the anatomical organ.

13. The surgical system according to claim 8, wherein, Identify at least one of the anatomical organs and anatomical structures based on the stored data.

14. The surgical system of claim 13, wherein, The stored data includes at least one of preoperative data, user preference data, and data from previously performed surgical procedures.

15. A surgical system for use with surgical instruments in surgical procedures, said surgical system comprising: At least one imaging device; and Control circuit, the control circuit being configured to: Identify the anatomical organ targeted by the surgical procedure; Identify anatomical structures related to the surgery based on visualization data from the at least one imaging device, wherein the visualization data includes structured light data and spectral data, wherein the structured light data is generated by the at least one imaging device based on sensing light patterns projected onto the surface of the anatomical organ, and wherein the anatomical structures are identified based on the anatomical contours of the anatomical organs contained in the visualization data and subsurface tissue features of the anatomical structures contained in the visualization data. Recommended surgical resection path for removing a portion of the anatomical organ using the surgical instruments, wherein the surgical resection path is determined based on the anatomical structure; and The surgical resection path was modified during the surgical procedure.

16. The surgical system of claim 15, wherein, The surgical resection path is superimposed on a virtual three-dimensional structure of at least a portion of the anatomical organ.

17. The surgical system of claim 15, wherein, Identifying the anatomical structure of the anatomical organ includes distinguishing the tissue type of the anatomical organ.

Images