Surgical system for overlaying surgical instrument data onto virtual three-dimensional construct of organ

Abstract

To provide a surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ.SOLUTION: A surgical system 100 comprises at least one imaging device 120 and a control circuit. The control circuit is configured to: identify anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device; propose a surgical resection path for removing a portion of the anatomical organ by the surgical instrument; and present parameters of the surgical instrument in accordance with the surgical resection path. The surgical resection path is determined based on the anatomical structures.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Surgical systems often incorporate imaging systems that can enable clinician(s) to view the surgical site and / or one or more portions thereof on one or more displays, such as, for example, a monitor. The display(s) can be local to the surgical theater and / or remote. The imaging system can include a scope with a camera that views the surgical site and transmits the view to a display viewable by the clinician. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngological-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. Imaging systems may be limited by the information they can recognize and / or convey to the clinician(s). For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognized during surgery by a particular imaging system. Additionally, certain imaging systems may not be able to communicate and / or convey certain information to the clinician(s) during surgery. Summary of the Invention [Means for solving the problem]

[0002] In one general aspect, a surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ is disclosed, the surgical system including at least one imaging device and control circuitry configured to: identify anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device; propose a surgical resection path for removing a portion of the anatomical organ with the surgical instrument; and present parameters of the surgical instrument in accordance with the surgical resection path. The surgical resection path is determined based on the anatomical structures.

[0003] In another general aspect, a surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ is disclosed, the surgical system including at least one imaging device and control circuitry configured to identify anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device, propose a surgical resection path for removing a portion of the anatomical organ with the surgical instrument, and adjust parameters of the surgical instrument in accordance with the surgical resection path. The surgical resection path is determined based on the anatomical structures.

[0004] In yet another general aspect, a surgical system for use with a surgical stapling instrument in a surgical procedure performed on an anatomical organ is disclosed, the surgical system including at least one imaging device and control circuitry configured to propose a surgical resection path for removing a portion of the anatomical organ with the surgical stapling instrument, propose a staple cartridge placement along the surgical resection path, and monitor tissue displacement along the proposed surgical resection path due to deployment of a staple line from the staple cartridge into tissue of the staple cartridge placement. [Brief explanation of the drawings]

[0005] The novel features of the various aspects are set forth with particularity in the appended claims. However, the described aspects, both as to organization and method of operation, may best be understood by reference to the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, in accordance with at least one embodiment of the present disclosure, configured to identify critical structures below a tissue surface. [Figure 2] FIG. 1 is a schematic diagram of a control system of a surgical visualization system, according to at least one aspect of the present disclosure. [Figure 2A]1 illustrates a control circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Figure 2B] 1 illustrates a combinational logic circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Figure 2C] 1 illustrates a sequential logic circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Figure 3] 2 is a schematic diagram illustrating triangulation between the surgical device, imaging device, and critical structure of FIG. 1 to determine the depth da of the critical structure below the tissue surface, according to at least one embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a surgical visualization system configured to identify critical structures below a tissue surface, according to at least one embodiment of the present disclosure, the surgical visualization system including a pulsed light source that determines the depth da of the critical structure below the tissue surface. [Figure 5] FIG. 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, in accordance with at least one embodiment of the present disclosure, configured to identify critical structures below a tissue surface. [Figure 6] FIG. 1 is a schematic diagram of a surgical visualization system including a three-dimensional camera configured to identify critical structures embedded within tissue, in accordance with at least one embodiment of the present disclosure. [Figure 7A] 7A and 7B are views of the critical structure captured by the 3D camera of FIG. 6, where FIG. 7A is a view from the left lens of the 3D camera and FIG. 7B is a view from the right lens of the 3D camera, according to at least one embodiment of the present disclosure. [Figure 7B] 7A and 7B are views of the critical structure captured by the 3D camera of FIG. 6, where FIG. 7A is a view from the left lens of the 3D camera and FIG. 7B is a view from the right lens of the 3D camera, according to at least one embodiment of the present disclosure. [Figure 8]7 is a schematic diagram of the surgical visualization system of FIG. 6 capable of determining a camera-to-critical structure distance dw from the three-dimensional camera to the critical structure, according to at least one embodiment of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of a surgical visualization system utilizing two cameras to determine the location of implanted critical structures, in accordance with at least one embodiment of the present disclosure. [Figure 10A] FIG. 1 is a schematic diagram of a surgical visualization system that utilizes a camera that is moved axially between multiple known positions to determine the location of an implanted critical structure, in accordance with at least one embodiment of the present disclosure. [Figure 10B] FIG. 10B is a schematic diagram of the surgical visualization system of FIG. 10A in which the camera is moved axially and rotationally between multiple known positions to determine the location of implanted critical structures, according to at least one embodiment of the present disclosure. [Figure 11] FIG. 1 is a schematic diagram of a control system of a surgical visualization system, according to at least one aspect of the present disclosure. [Figure 12] FIG. 1 is a schematic diagram of a structured light source of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Figure 13A] FIG. 10 shows a graph of absorption coefficient versus wavelength for various biomaterials, in accordance with at least one embodiment of the present disclosure. [Figure 13B] FIG. 1 is a schematic illustration of visualization of an anatomical structure with a spectral surgical visualization system, in accordance with at least one aspect of the present disclosure. [Figure 13C] 13A-13E show exemplary hyperspectral specific signatures for differentiating anatomical structures from occlusions, in accordance with at least one embodiment of the present disclosure, where FIG. 13C is a graphical representation of a ureteral signature versus occlusions, FIG. 13D is a graphical representation of an arterial signature versus occlusions, and FIG. 13E is a graphical representation of a neural signature versus occlusions. [Figure 13D]13A-13E show exemplary hyperspectral specific signatures for differentiating anatomical structures from occlusions, in accordance with at least one embodiment of the present disclosure, where FIG. 13C is a graphical representation of a ureteral signature versus occlusions, FIG. 13D is a graphical representation of an arterial signature versus occlusions, and FIG. 13E is a graphical representation of a neural signature versus occlusions. [Figure 13E] 13A-13E show exemplary hyperspectral specific signatures for differentiating anatomical structures from occlusions, in accordance with at least one embodiment of the present disclosure, where FIG. 13C is a graphical representation of a ureteral signature versus occlusions, FIG. 13D is a graphical representation of an arterial signature versus occlusions, and FIG. 13E is a graphical representation of a neural signature versus occlusions. [Figure 14] 1 is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense distance to a critical anatomical structure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) located on a common device, in accordance with at least one embodiment of the present disclosure. [Figure 15] 17B is a schematic diagram of an emitted wave, a received wave, and a delay between the emitted and received waves of the NIR time-of-flight measurement system of FIG. 17A in accordance with at least one embodiment of the present disclosure. FIG. [Figure 16] 1 illustrates an NIR time-of-flight measurement system configured to sense distances to different structures, according to at least one embodiment of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) on separate devices. [Figure 17] FIG. 1 is a block diagram of a computer-implemented interactive surgical system in accordance with at least one aspect of the present disclosure. [Figure 18] FIG. 1 is a diagram of a surgical system used to perform a surgical procedure in an operating room, according to at least one aspect of the present disclosure. [Figure 19] 1 illustrates a computer-implemented interactive surgical system in accordance with at least one aspect of the present disclosure. [Figure 20] 1 shows a diagram of a situation-aware surgical system according to at least one aspect of the present disclosure. [Figure 21] 1 shows a timeline illustrating situational awareness of a hub, in accordance with at least one aspect of the present disclosure. [Figure 22] FIG. 10 is a process logic flow diagram illustrating a control program or logic configuration for correlating visualization data and instrument data in accordance with at least one aspect of the present disclosure. [Figure 23] 1 is a schematic diagram of a surgical instrument according to at least one embodiment of the present disclosure. [Figure 24] FIG. 1 is a graph showing a composite data set and virtual gauges of Force-to-Close ("FTC") and Force-to-Fire ("FTF"), in accordance with at least one embodiment of the present disclosure. [Figure 25A] 1 illustrates a standard view of a visualization system screen displaying a live feed of an end effector in a surgical field, according to at least one aspect of the present disclosure. [Figure 25B] 1 illustrates an expanded view of a visualization system screen displaying a live feed of an end effector in a surgical field, according to at least one aspect of the present disclosure. [Figure 26] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for synchronizing movement of a virtual representation of an end effector component with the actual movement of the end effector component, in accordance with at least one aspect of the present disclosure. [Figure 27] In accordance with at least one embodiment of the present disclosure, a diagram showing anatomical structures in a body wall and a cavity below the body wall, with a trocar passing through the body wall and entering the cavity, and a screen displaying the distance of the trocar from the anatomical structures, the risks associated with presenting surgical instruments through the trocar, and the associated estimated surgical time. [Figure 28] 1 illustrates a virtual three-dimensional ("3D") structure of a stomach exposed to structured light from a structured light projector, according to at least aspects of the present disclosure. [Figure 29]FIG. 1 is a logic flow diagram of a process illustrating a control program or logic configuration for correlating visualization data and instrument data in accordance with at least one aspect of the present disclosure, with dashed boxes indicating alternative implementations of the process. [Figure 30] 1 illustrates a virtual 3D structure of a stomach exposed to structured light from a structured light projector, according to at least aspects of the present disclosure. [Figure 31] FIG. 1 is a logic flow diagram of a process illustrating a control program or logic configuration for proposing a resection path for removing a portion of an anatomical organ, in accordance with at least one aspect of the present disclosure, where dashed boxes indicate alternative implementations of the process. [Figure 32A] 1 illustrates a live view of a surgical field on a screen of a visualization system at the start of a surgical procedure, in accordance with at least one aspect of the present disclosure. [Figure 32B] FIG. 32B is an enlarged view of a portion of the surgical field of FIG. 32A outlining a proposed surgical resection path superimposed on the surgical field, according to at least one embodiment of the present disclosure. [Figure 32C] 32C shows a live view of the surgical field of FIG. 32B 43 minutes after the start of the surgical procedure, according to at least one embodiment of the present disclosure. [Figure 32D] 32D shows an enlarged view of the surgical field of FIG. 32C outlining a proposed surgical resection path modification in accordance with at least one embodiment of the present disclosure. [Figure 33] FIG. 1 is a logic flow diagram of a process showing a control program or logic configuration for presenting surgical instrument parameters on or near a proposed surgical resection path in accordance with at least one aspect of the present disclosure, with dashed boxes indicating alternative implementations of the process. [Figure 34] 1 illustrates a virtual 3D structure of a patient's stomach undergoing sleeve gastrectomy, in accordance with at least one embodiment of the present disclosure. [Figure 35] 35 shows a completed virtual resection of the stomach of FIG. 34. [Figure 36A] 1 illustrates the firing of a surgical stapling instrument in accordance with at least one aspect of the present disclosure. [Figure 36B]1 illustrates the firing of a surgical stapling instrument in accordance with at least one aspect of the present disclosure. [Figure 36C] 1 illustrates the firing of a surgical stapling instrument in accordance with at least one aspect of the present disclosure. [Figure 37] FIG. 10 is a process logic flow diagram illustrating a control program or logic configuration for adjusting the firing rate of a surgical instrument, in accordance with at least one aspect of the present disclosure. [Figure 38] FIG. 10 is a logic flow diagram of a process illustrating a control program or logic configuration for proposed staple cartridge placement along a proposed surgical resection path in accordance with at least one aspect of the present disclosure. [Figure 39] FIG. 1 is a process logic flow diagram illustrating a control program or logic configuration for suggesting surgical resection of a portion of an organ, in accordance with at least one aspect of the present disclosure. [Figure 40] FIG. 1 is a logic flow diagram of a process illustrating a control program or logic configuration for estimating the volume loss of an organ resulting from removal of a selected portion of the organ, according to at least one embodiment of the present disclosure. [Figure 41A] 1 illustrates a patient's lungs exposed to structured light, including a portion to be removed during surgery, in accordance with at least one embodiment of the present disclosure. [Figure 41B] 41B illustrates the lung of the patient of FIG. 41A after a portion is removed, in accordance with at least one embodiment of the present disclosure. [Figure 41C] 41A and 41B, showing graphs measuring maximum lung capacity of the patient's lungs before and after resection of a portion of the lung, in accordance with at least one embodiment of the present disclosure. [Figure 42] 1 shows a graph measuring the partial pressure of carbon dioxide ("PCO2") in a patient's lungs before, immediately after, and one minute after removing a portion of the lung, according to at least one embodiment of the present disclosure. [Figure 43] FIG. 1 is a process logic flow diagram illustrating a control program or logic configuration for detecting tissue abnormalities using visualized and non-visualized data, in accordance with at least one aspect of the present disclosure. [Figure 44A] 1 illustrates the right lung in a first state with an imaging device illuminating a light pattern on its surface, according to at least one aspect of the present disclosure. [Figure 44B] 45 illustrates the right lung of FIG. 44 in a second state with an imaging device illuminating a light pattern on its surface, in accordance with at least one embodiment of the present disclosure. [Figure 44C] 44B shows the apex of the right lung of FIG. 44A in accordance with at least one embodiment of the present disclosure. [Figure 44D] 44C shows the apex of the right lung of FIG. 44B in accordance with at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0006] The applicant of the present application owns the following concurrently filed US patent applications, each of which is incorporated herein by reference in its entirety: · Attorney Docket No. END9228USNP1 / 190580-1M, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY"; · Attorney Docket No. END9227USNP1 / 190579-1, entitled "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM"; · Attorney Docket No. END9226USNP1 / 190578-1, entitled "SURGICAL SYSTEM CONTROL BASED ON MULTIPLE SENSED PARAMETERS"; · Attorney Docket No. END9225USNP1 / 190577-1, entitled "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS"; · Attorney Docket No. END9224USNP1 / 190576-1, entitled "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS"; · Attorney Docket No. END9223USNP1 / 190575-1, entitled "SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA"; · Attorney Docket No. END9222USNP1 / 190574-1, entitled "SURGICAL SYSTEMS FOR GENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED"; · Attorney Docket No. END9220USNP1 / 190572-1, entitled "SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS"; · Attorney Docket No. END9219USNP1 / 190571-1, entitled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE"; · Attorney Docket No. END9218USNP1 / 190570-1, entitled "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT"; · Attorney Docket No. END9217USNP1 / 190569-1, entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS"; · Attorney Docket No. END9216USNP1 / 190568-1, entitled "Analyzing Surgical Trends by a Surgical System."

[0007] The applicant of this application owns the following U.S. patent applications, filed on March 15, 2019, each of which is incorporated herein by reference in its entirety: · U.S. Patent Application No. 16 / 354,417, entitled "INPUT CONTROLS FOR ROBOTIC SURGERY"; · U.S. Patent Application No. 16 / 354,420, entitled "DUAL MODE CONTROLS FOR ROBOTIC SURGERY"; · U.S. Patent Application No. 16 / 354,422, entitled "MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY"; · U.S. Patent Application No. 16 / 354,440, entitled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY"; · U.S. Patent Application No. 16 / 354,444, entitled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE"; · U.S. Patent Application No. 16 / 354,454, entitled "ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS"; · U.S. Patent Application No. 16 / 354,461, entitled "SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS"; · U.S. Patent Application No. 16 / 354,470, entitled "SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS"; · U.S. Patent Application No. 16 / 354,474, entitled "ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES"; U.S. Patent Application No. 16 / 354,478, entitled "ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK," and · U.S. Patent Application No. 16 / 354,481, entitled "JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS."

[0008] The applicant of the present application also owns the following U.S. patent applications, filed on September 11, 2018, the entire contents of each of which are incorporated herein by reference: · U.S. Patent Application No. 16 / 128,179, entitled "SURGICAL VISUALIZATION PLATFORM"; · U.S. Patent Application No. 16 / 128,180, entitled "CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE"; · U.S. Patent Application No. 16 / 128,198, entitled "SINGULAR EMR SOURCE EMITTER ASSEMBLY"; · U.S. Patent Application No. 16 / 128,207, entitled "COMBINATION EMITTER AND CAMERA ASSEMBLY"; · U.S. Patent Application No. 16 / 128,176, entitled "SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES"; · U.S. Patent Application No. 16 / 128,187, entitled "SURGICAL VISUALIZATION OF MULTIPLE TARGETS"; · U.S. Patent Application No. 16 / 128,192, entitled "VISUALIZATION OF SURGICAL DEVICES"; · U.S. Patent Application No. 16 / 128,163, entitled "OPERATIVE COMMUNICATION OF LIGHT"; · U.S. Patent Application No. 16 / 128,197, entitled "ROBOTIC LIGHT PROJECTION TOOLS"; · U.S. Patent Application No. 16 / 128,164, entitled "SURGICAL VISUALIZATION FEEDBACK SYSTEM"; · U.S. Patent Application No. 16 / 128,193, entitled "SURGICAL VISUALIZATION AND MONITORING"; · U.S. Patent Application No. 16 / 128,195, entitled "INTEGRATION OF IMAGING DATA"; · U.S. Patent Application No. 16 / 128,170, entitled "ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS"; · U.S. Patent Application No. 16 / 128,183, entitled "SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS"; · U.S. Patent Application No. 16 / 128,172, entitled "ROBOTIC SYSTEM WITH SEPARATE PHOTOACOUSTIC RECEIVER"; · U.S. Patent Application No. 16 / 128,185, entitled "FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION."

[0009] The applicant of the present application also owns the following U.S. patent applications, filed March 29, 2018, the entire contents of each of which are incorporated herein by reference: · U.S. Patent Application No. 15 / 940,627, now U.S. Patent Application Publication No. 2019 / 0201111, entitled "DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS"; · U.S. Patent Application No. 15 / 940,676, now U.S. Patent Application Publication No. 2019 / 0201142, entitled "AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS"; · U.S. Patent Application No. 15 / 940,711, now U.S. Patent Application Publication No. 2019 / 0201120, entitled "SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS"; · U.S. Patent Application No. 15 / 940,722, now U.S. Patent Application Publication No. 2019 / 0200905, entitled "CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY."

[0010] The applicant of the present application owns the following U.S. patent applications, filed December 4, 2018, the disclosures of each of which are incorporated herein by reference in their entirety: · U.S. Patent Application No. 16 / 209,395, now U.S. Patent Application Publication No. 2019 / 0201136, entitled "METHOD OF HUB COMMUNICATION"; · U.S. Patent Application No. 16 / 209,403, now U.S. Patent Application Publication No. 2019 / 0206569, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB"; · U.S. Patent Application No. 16 / 209,407, now U.S. Patent Application Publication No. 2019 / 0201137, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL"; · U.S. Patent Application No. 16 / 209,416, now U.S. Patent Application Publication No. 2019 / 0206562, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS"; · U.S. Patent Application No. 16 / 209,423, now U.S. Patent Application Publication No. 2019 / 0200981, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS"; · U.S. Patent Application No. 16 / 209,427, now U.S. Patent Application Publication No. 2019 / 0208641, entitled "METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES"; · U.S. Patent Application No. 16 / 209,433, now U.S. Patent Application Publication No. 2019 / 0201594, entitled "METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB"; · U.S. Patent Application No. 16 / 209,447, now U.S. Patent Application Publication No. 2019 / 0201045, entitled "METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB"; · U.S. Patent Application No. 16 / 209,453, now U.S. Patent Application Publication No. 2019 / 0201046, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES"; · U.S. Patent Application No. 16 / 209,458, now U.S. Patent Application Publication No. 2019 / 0201047, entitled "METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE"; · U.S. Patent Application No. 16 / 209,465, now U.S. Patent Application Publication No. 2019 / 0206563, entitled "METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION"; · U.S. Patent Application No. 16 / 209,478, now U.S. Patent Application Publication No. 2019 / 0104919, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE"; · U.S. Patent Application No. 16 / 209,490, now U.S. Patent Application Publication No. 2019 / 0206564, entitled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION"; · U.S. Patent Application No. 16 / 209,491, now U.S. Patent Application Publication No. 2019 / 0200998, entitled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS."

[0011] Before describing various aspects of the surgical visualization platform in detail, it should be noted that the exemplary embodiments are not limited in application or use to the details of construction and arrangement of parts illustrated in the accompanying drawings and specification. The exemplary embodiments may be implemented in or incorporated into other aspects, variations, and modifications, and may be practiced or carried out in various ways. Furthermore, unless otherwise specified, the terms and phrases employed herein have been chosen for the convenience of the reader for the purpose of describing the exemplary embodiments, and not for the purpose of limiting them. Furthermore, it should be understood that one or more of the aspects, aspect expressions, and / or examples described below can be combined with any one or more of the other aspects, aspect expressions, and / or examples described below.

[0012] Surgical Visualization System The present disclosure is directed to a surgical visualization platform that utilizes "digital surgery" to obtain additional information regarding a patient's anatomy and / or a surgical procedure. The surgical visualization platform is further configured to communicate data and / or information to one or more clinicians in a useful manner. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomy and / or a surgical procedure.

[0013] "Digital surgery" may encompass robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and the like. While the various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, the surgical visualization platforms are not limited to use with robotic surgical systems. In certain examples, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0014] In certain examples, surgical systems incorporating surgical visualization platforms may enable smart dissection to identify and avoid critical structures. Critical structures include anatomical structures, such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors, among other anatomical structures. In other examples, critical structures may be foreign structures in an anatomical region, such as surgical devices, surgical fasteners, clips, clasps, bougies, bands, and / or plates. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Exemplary critical structures are described further herein. Smart dissection may provide improved intraoperative guidance for dissection and / or enable smarter decisions, for example, through critical anatomical structure detection and avoidance techniques.

[0015] Surgical systems incorporating surgical visualization platforms may also enable smart anastomosis, providing more consistent anastomoses at optimal location(s) through improved workflow. Cancer localization methods may also be improved by the various surgical visualization platforms and procedures described herein. For example, cancer localization methods can identify and track the location, orientation, and margins of a cancer. In certain instances, cancer localization methods can compensate for movement of instruments, the patient, and / or the patient's anatomy during surgery and provide guidance returning the clinician to the point of interest.

[0016] In certain aspects of the present disclosure, the surgical visualization platform may provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health without the need for physical touch, particularly during incision and / or placement of a stapling device within the tissue. Certain tissue characterization techniques described herein can be utilized without the use of ionizing radiation and / or contrast agents. With regard to lymph node diagnosis and mapping, the surgical visualization platform can pre-operatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0017] During a surgical procedure, information available to a clinician via the "naked eye" and / or an imaging system may provide an incomplete view of the surgical site. For example, certain structures, such as structures embedded or buried within an organ, may be at least partially hidden from view, i.e., invisible. Additionally, certain dimensions and / or relative distances may be difficult to ascertain with existing sensor systems and / or difficult to grasp with the "naked eye." Furthermore, certain structures may move pre-operatively (e.g., before surgery but after a pre-operative scan) and / or during surgery. In such instances, a clinician may not be able to accurately determine the location of critical structures during surgery.

[0018] Uncertainty about the location of critical structures and / or the proximity between critical structures and surgical tools can hinder a clinician's decision-making process. For example, a clinician may avoid certain areas to avoid inadvertently dissecting critical structures. However, the avoided areas may be unnecessarily large and / or at least partially misplaced. Due to uncertainty and / or excessive / overcaution, a clinician may be unable to reach a particular desired area. For example, even if a critical structure is not in that particular area and / or the clinician's actions in that particular area would have no adverse effects, an overly cautious clinician may leave portions of tumor and / or other undesirable tissue in an effort to avoid the critical structure. In certain instances, surgical outcomes may improve with increased knowledge and / or certainty, allowing the surgeon to be more precise and, in certain instances, less / more aggressive in certain anatomical areas.

[0019] In various aspects, the present disclosure provides a surgical visualization system for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what a clinician sees with the "naked eye" and / or what an imaging system can recognize and / or communicate to a clinician. The various surgical visualization systems can augment and enhance what a clinician can know prior to tissue treatment (e.g., incision), thereby improving outcomes in various instances.

[0020] For example, a visualization system can include a first light emitter configured to emit a plurality of spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to the spectral waves (spectral imaging), and / or the light pattern. Note that throughout the following disclosure, unless specifically referring to visible light, all references to "light" can include electromagnetic waves or photons in the visible and / or non-visible portions of the electromagnetic radiation (EMR) wavelength spectrum. The surgical visualization system can also include an imaging system and control circuitry in signal communication with the receiver(s) and the imaging system. Based on output from the receiver(s), the control circuitry can determine a geometric surface map, i.e., a three-dimensional surface topography, of visible surfaces at the surgical site and one or more distances relative to the surgical site. In certain examples, the control circuitry can determine one or more distances to at least partially hidden structures. Furthermore, the imaging system can communicate the geometric surface map and one or more distances to a clinician. In such instances, the augmented view of the surgical site provided to the clinician can provide a display of the hidden structures within a context relevant to the surgical site. For example, the imaging system can virtually enhance the hidden structures on a geometric surface map of the obscuring and / or occluding tissue, similar to a line drawn on the ground to indicate utility lines below a surface. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to the visible occluding tissue and / or to the at least partially obscured structures, and / or the depth of the hidden structures below the visible surface of the occluding tissue. For example, the visualization system can determine the distance to the augmented line on the surface of the visible tissue and communicate that distance to the imaging system.

[0021] In various aspects of the present disclosure, a surgical visualization system for intraoperative identification and avoidance of critical structures is disclosed. Such a surgical visualization system can provide valuable information to a clinician during a surgical procedure. As a result, the clinician can maintain momentum throughout the surgical procedure while knowing that the surgical visualization system is tracking critical structures, such as the ureter, specific nerves, and / or critical blood vessels, that may be accessible during the incision. In one aspect, the surgical visualization system can provide instructions to the clinician in sufficient time to pause and / or slow down the surgical procedure and assess the proximity of critical structures to prevent inadvertent damage to the structures. The surgical visualization system can provide the clinician with an ideal, optimized, and / or customizable amount of information to enable the clinician to move steadily and / or quickly across the structure while avoiding inadvertent damage to healthy tissue and / or critical structure(s), thereby minimizing the risk of injury resulting from the surgical procedure.

[0022] FIG. 1 is a schematic diagram of a surgical visualization system 100 according to at least one embodiment of the present disclosure. The surgical visualization system 100 can generate a visual display of a critical structure 101 within an anatomical region. The surgical visualization system 100 can be used, for example, in clinical analysis and / or medical intervention. In certain examples, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near-real-time information regarding proximity, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured for intraoperative identification of a critical structure(s) and / or to facilitate avoidance of the critical structure(s) 101 by a surgical device. For example, by identifying the critical structure 101, a clinician can avoid manipulating a surgical device around the critical structure 101 and / or a predetermined area proximal to the critical structure 101 during a surgical procedure. The clinician can avoid dissections around, for example, veins, arteries, nerves, and / or blood vessels identified as the critical structure 101. In various examples, the critical structures 101 may be determined on a patient-by-patient and / or procedure-by-procedure basis.

[0023] The surgical visualization system 100 incorporates tissue identification and geometric surface mapping in combination with a distance sensor system 104. In combination, these features of the surgical visualization system 100 can determine the location of a critical structure 101 within an anatomical region and / or the proximity of a surgical device 102 to a visible tissue surface 105 and / or critical structure 101. Additionally, the surgical visualization system 100 includes an imaging system including an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various examples, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectrum camera) configured to detect reflected spectral waveforms and generate a spectral cube of an image based on molecular responses to various wavelengths. The view from the imaging device 120 can be provided to a clinician and, in various aspects of the present disclosure, can be enhanced with additional information based on the tissue identification, landscape mapping, and distance sensor system 104. In such an example, the surgical visualization system 100 includes multiple subsystems, namely, an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem, which can work together to provide advanced data synthesis and integrated information to the clinician(s) during surgery.

[0024] The imaging device may include, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In various aspects of the present disclosure, the imaging system may include an imaging device such as, for example, an endoscope. Additionally or alternatively, the imaging system may include an imaging device such as, for example, an arthroscope, angioscope, bronchoscope, cholangioscope, colonoscope, cystoscope, duodenoscope, enteroscope, esophagogastroduodenoscope (gastroscope), laryngoscope, nasopharyngological-nephroscope, sigmoidoscope, thoracoscope, ureteroscope, or exoscope. In other examples, such as open surgery applications, the imaging system may not include a scope.

[0025] In various aspects of the present disclosure, the tissue characterization subsystem can be achieved using a spectral imaging system. The spectral imaging system can rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled "System and method for gross anatomic pathology using hyperspectral imaging," issued March 1, 2016, and incorporated herein by reference in its entirety.

[0026] In various aspects of the present disclosure, the surface mapping subsystem can be accomplished using a light pattern system, as further described herein. The use of light patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be utilized in the surgical visualization systems described herein.

[0027] Structured light is the process of projecting a known pattern (often a grid or horizontal bars) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, published March 2, 2017, entitled "SET COMPRISING A SURGICAL INSTRUMENT," and U.S. Patent Application Publication No. 2017 / 0251900, published September 7, 2017, entitled "DEPICTION SYSTEM," disclose surgical systems that include a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, published March 2, 2017, entitled "SET COMPRISING A SURGICAL INSTRUMENT," and U.S. Patent Application Publication No. 2017 / 0251900, published September 7, 2017, entitled "DEPICTION SYSTEM," are incorporated herein by reference in their entirety.

[0028] In various aspects of the present disclosure, a distance determination system can be incorporated into a surface mapping system. For example, structured light can be used to generate a three-dimensional virtual model of the visible surface and determine various distances to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to identified tissues (or other structures) at the surgical site.

[0029] FIG. 2 is a schematic diagram of a control system 133 that can be utilized with surgical visualization system 100. Control system 133 includes control circuitry 132 in signal communication with memory 134. Memory 134 stores instructions executable by control circuitry 132 for determining and / or recognizing a structure of interest (e.g., structure of interest 101 of FIG. 1 ), determining and / or calculating one or more distances and / or three-dimensional digital representations, and communicating certain information to one or more clinicians. For example, memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. Control system 133 also includes an imaging system 142 having one or more cameras 144 (such as imaging device 120 of FIG. 1 ), one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 can include one or more image sensors 135 (e.g., visible light, spectral imagers, three-dimensional lenses, among others) for receiving signals from various light sources emitting light in various visible and invisible spectrums. The display 146 can include one or more screens or monitors for portraying real, virtual, and / or virtually augmented images and / or information to one or more clinicians.

[0030] In various embodiments, the heart of the camera 144 is the image sensor 135. Modern image sensors 135 are typically solid-state electronic devices containing up to millions of discrete photodetector sites called pixels. Image sensor 135 technologies fall into one of two categories: charge-coupled device (CCD) imagers and complementary metal-oxide semiconductor (CMOS) imagers, with short-wave infrared (SWIR) imaging being a more recent emerging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCMOS"), consisting of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 are sensitive to wavelengths between approximately 350 and 1050 nm, although this range is typically between 400 and 1000 nm. CMOS sensors generally have greater sensitivity to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish between colors. Thus, there are two types of color CCD cameras: one-chip and three-chip. One-chip color CCD cameras offer a commonly employed low-cost imaging solution, using a mosaic (e.g., Bayer) optical filter to separate the incoming light into a series of colors and employ an interpolation algorithm to resolve a full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras offer higher resolution by employing a prism, directing each section of the incoming spectrum to a different chip. Because each point in the object's space has a distinct RGB intensity value rather than using an algorithm to determine color, more accurate color reproduction is possible. Three-chip cameras offer significantly higher resolution.

[0031] The control system 133 also includes a spectral light source 150 and a structured light source 152. In certain examples, a single light source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, the single light source can be pulsed to provide light within the visible spectrum (e.g., infrared spectrum light) and wavelengths of light above the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectrum light source. In various examples, the tissue identification logic 140 can identify the structure(s) of interest via data from the spectral light source 150 received by the image sensor 135 portion of the camera 144. The surface mapping logic 136 can determine the contours of the surface of the visible tissue based on the reflected structured light. Through time-of-flight measurements, the distance determination logic 141 can determine one or more distances to the visible tissue and / or structure of interest 101. One or more outputs from the surface mapping logic 136, tissue identification logic 140, and distance determination logic 141 may be provided to the imaging logic 138 and combined, integrated, and / or superimposed for transmission to the clinician via the display 146 of the imaging system 142.

[0032] 2A-2C, various aspects of the control circuitry 132 for controlling various aspects of the surgical visualization system 100 are now briefly described. Referring to FIG. 2A, a control circuitry 400 configured to control aspects of the surgical visualization system 100 is illustrated, in accordance with at least one embodiment of the present disclosure. The control circuitry 400 can be configured to implement various processes described herein. The control circuitry 400 may comprise a microcontroller including one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuitry 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute the machine instructions to implement the various processes described herein. The processor 402 may be any one of a number of single-core or multi-core processors known in the art. The memory circuitry 404 can include volatile and non-volatile storage media. The processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuitry 404 of the present disclosure.

[0033] 2B illustrates a combinational logic circuit 410 configured to control aspects of the surgical visualization system 100, in accordance with at least one aspect of the present disclosure. The combinational logic circuit 410 can be configured to implement various processes described herein. The combinational logic circuit 410 may comprise a finite state machine comprising combinational logic 412 configured to receive data associated with a surgical instrument or tool at an input 414, process the data through the combinational logic 412, and provide an output 416.

[0034] FIG. 2C illustrates a sequential logic circuit 420 configured to control aspects of the surgical visualization system 100 in accordance with at least one embodiment of the present disclosure. The sequential logic circuit 420 or the combinatorial logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 may comprise a finite state machine. The sequential logic circuit 420 may comprise, for example, the combinatorial logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store the current state of the finite state machine. In certain examples, the sequential logic circuit 420 may be synchronous or asynchronous. The combinatorial logic 422 is configured to receive data associated with a surgical device or system from an input 426, process the data through the combinatorial logic 422, and provide an output 428. In other embodiments, a circuit may comprise a combination of a processor (e.g., processor 402 of FIG. 2A ) and a finite state machine that implements various processes herein. In other aspects, the finite state machine may comprise a combination of combinational logic (eg, combinational logic 410 of FIG. 2B) and sequential logic 420.

[0035] Referring again to the surgical visualization system 100 of FIG. 1 , the critical structure 101 may be an anatomical structure of interest. For example, the critical structure 101 may be a ureter, an artery such as the superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor, among other anatomical structures. In other examples, the critical structure 101 may be a foreign body structure in the anatomical region, such as a surgical device, a surgical fastener, a clip, a clasp, a bougie, a band, and / or a plate. Exemplary critical structures are further described herein and in the aforementioned U.S. patent applications, including, for example, U.S. Patent Application No. 16 / 128,192, entitled "VISUALIZATION OF SURGICAL DEVICES," filed September 11, 2018, each of which is incorporated herein by reference in its entirety.

[0036] In one aspect, the structure of interest 101 may be embedded in the tissue 103. In other words, the structure of interest 101 may be located below the surface 105 of the tissue 103. In such an example, the tissue 103 obscures the structure of interest 101 from the view of the clinician. The structure of interest 101 is also occluded from the view of the imaging device 120 by the tissue 103. The tissue 103 may be, for example, fat, connective tissue, adhesions, and / or organs. In other examples, the structure of interest 101 may be partially occluded from view.

[0037] 1 also illustrates a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from the distal end of the shaft of the surgical device 102. The surgical device 102 may be any suitable surgical device, such as, for example, a dissector, a stapler, a grasper, a clip applier, and / or an energy device, including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector. Additionally or alternatively, the surgical device 102 may include another imaging or diagnostic modality, such as, for example, an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 may be configured to identify one or more critical structures 101 and achieve access of the surgical device 102 to the critical structure(s) 101.

[0038] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral camera, a multispectral camera, or a selective spectrum camera, as described further herein. The imaging device 120 may also include a waveform sensor 122 (e.g., a spectral imaging sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens used together to generate a three-dimensional image of the surgical site by simultaneously recording two two-dimensional images, render a three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images indicative of the topography of visible tissue and the identification and location of hidden critical structures, as described further herein. For example, as shown in FIG. 1, the field of view of the imaging device 120 can be overlaid with a light pattern (structured light) on the surface 105 of the tissue.

[0039] In one aspect, the surgical visualization system 100 may be incorporated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include a rigid structural member 116 and a joint 118, which may include servo motor control. The first robotic arm 112 is configured to manipulate the surgical device 102, and the second robotic arm 114 is configured to manipulate the imaging device 120. A robotic control unit may be configured to issue control movements to the robotic arms 112, 114 that may act on, for example, the surgical device 102 and the imaging device 120.

[0040] The surgical visualization system 100 also includes an emitter 106 configured to emit a light pattern, such as stripes, grid lines, and / or dots, to enable determination of the topography or landscape of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from an emitter 106 located, for example, on the surgical device 102 and / or one of the robotic arms 112, 114 and / or the imaging device 120. In one aspect, the projected light array 130 is employed to determine the surface 105 of the tissue 103 and / or the shape defined by movement of the surface 105 during surgery. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances relative to the surface 105.

[0041] In one aspect, the imaging device 120 may also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach the structure of interest 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effect of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted from the optical waveform emitter 123 may be configured to enable identification of a type of anatomical structure and / or body structure, such as the structure of interest 101. Identification of the structure of interest 101 may be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be tunable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectrum imaging system. In other examples, the waveform sensor 122 and the optical waveform emitter 123 can include, for example, an optoacoustic imaging system. In other examples, the optical waveform emitter 123 can be located on a surgical device separate from the imaging device 120.

[0042] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one embodiment, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system including an emitter, such as emitter 106, and a receiver 108, which may be disposed on the surgical device 102. In other examples, the time-of-flight emitter may be separate from the structured light emitter. In one general embodiment, the emitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a matching sensor. The time-of-flight distance sensor system 104 can detect the "time of flight," i.e., the time it takes for laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. The use of a very narrow light source in the emitter 106 allows the distance sensor system 104 to determine the distance to a surface 105 of the tissue 103 directly in front of the distance sensor system 104. With further reference to FIG. 1 , d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the tissue surface 105. The distance sensor system 104 is employed to measure the emitter-tissue distance d e The device-tissue distance d can be determined. t can be obtained from the known position of the emitter 106 on the shaft of the surgical device 102 relative to the distal end of the surgical device 102. In other words, if the distance between the emitter 106 and the distal end of the surgical device 102 is known, then the device-tissue distance d t is the emitter-tissue distance d e In certain examples, the shaft of the surgical device 102 can include one or more articulating joints and can be articulated relative to the emitter 106 and jaws. The articulating configuration can include, for example, an articulated vertebra-like structure. In certain examples, a three-dimensional camera can be utilized to triangulate one or more distances to the surface 105.

[0043] In various examples, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on a separate surgical device instead of the surgical device 102. For example, the receiver 108 can be mounted on a cannula or trocar through which the surgical device 102 extends to reach the surgical site. In yet other examples, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on another robotically controlled arm (e.g., robotic arm 114), on another robotically controlled movable arm, and / or on an operating room (OR) table or fixture. In certain examples, the imaging device 120 includes a time-of-flight receiver 108 that uses a line between the emitter 106 on the surgical device 102 and the imaging device 120 to determine the distance from the emitter 106 to the surface 105 of the tissue 103. For example, based on the known positions of the emitter 106 of the time-of-flight distance sensor system 104 (on the surgical device 102) and the receiver 108 (on the imaging device 120), the distance d e The three-dimensional position of the receiver 108 can be known and / or aligned with respect to the robot coordinate plane during surgery.

[0044] In certain examples, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by a first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by a second robotic arm 114. In other examples, the surgical visualization system 100 can be utilized separately from a robotic system. In such examples, the distance sensor system 104 can be independent of the robotic system.

[0045] In certain examples, one or more of the robotic arms 112, 114 may be separate from the main robotic system used in a surgical procedure. At least one of the robotic arms 112, 114 may be positioned and aligned to a particular coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 may control and / or align the position of the robotic arm(s) 112, 114 relative to a particular coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 may be aligned to a particular coordinate system.

[0046] Further referring to FIG. 1, d w is the camera-to-structure-of-interest distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the structure of interest 101, and d A is the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the critical structure 101 and the portion of the surface 105 closest to the surgical device 102). In various embodiments, the time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 is determined by the camera-to-critical structure distance d w The use of spectral imaging in combination with a time-of-flight sensor is further described herein. Further, referring now to FIG. 3, in various aspects of the present disclosure, the depth d of the critical structure 101 relative to the surface 105 of the tissue 103 is determined. A is the distance d w , and the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and thus the distance d between them). x ) is determined by triangulation, and the distance d e and d A The distance d is the sum of y can be determined.

[0047] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or range of waveforms) can be used to determine the camera-to-structure-of-interest distance d w can be determined, and a second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such an example, a different waveform can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0048] Additionally or alternatively, in certain instances, the distance d A can be determined from ultrasound, registered magnetic resonance views (MRI), or computed tomography (CT) scans. A can be determined by spectral imaging because the detected signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detected signal in a first way or by a first amount, while collagen can reduce the detected signal in a different second way or by a second amount.

[0049] 4, surgical device 162 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect a reflected waveform. Optical waveform emitter 123 is positioned a distance d from a common device, such as surgical device 162, as described further herein. t , and d w In such an example, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 is A can be determined as follows: d A =d w -d t .

[0050] As disclosed herein, various information regarding visible tissue, embedded vital structures, and surgical devices can be determined by utilizing approaches incorporating one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays in combination with an image sensor configured to detect spectral wavelengths and structured light arrays. Additionally, the image sensor can be configured to receive visible light to provide an image of the surgical site to the imaging system. Logic or algorithms are employed to distinguish between the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various examples, the imaging device 120 can include multiple image sensors.

[0051] Camera-critical structure distance d w can also be detected by one or more alternative methods. In one embodiment, for example, fluoroscopic visualization techniques, such as fluorescent indocyanine green (ICG), can be utilized to illuminate the critical structure 201 as shown in FIGS. 6-8. The camera 220 can include two optical waveform sensors 222, 224 that simultaneously capture left and right images of the critical structure 201 (FIGS. 7A and 7B). In such an example, the camera 220 can depict the emission of the critical structure 201 below the surface 205 of the tissue 203, at a distance d w can be determined by the known distance between sensors 222 and 224. In certain examples, distance can be more accurately determined by utilizing more than one camera or by moving a camera between multiple positions. In certain embodiments, one camera can be controlled by a first robotic arm and a second camera by another robotic arm. In such a robotic system, one camera can be a driven camera, for example, on a driven arm. The driven arm and the camera thereon can be programmed, for example, to track the other camera and maintain a particular distance and / or lens angle.

[0052] In yet another embodiment, the surgical visualization system 100 employs two separate waveform receivers (i.e., cameras / image sensors) to w 9, if a critical structure 301 or its contents (e.g., a blood vessel or the contents of a blood vessel) can emit a signal 302, such as by fluoroscopy, the actual location can be triangulated from two separate cameras 320a, 320b at known locations.

[0053] 10A and 10B, in another embodiment, a surgical visualization system employs a dithering or moving camera 440, distance d wmay be determined. The camera 440 is robotically controlled so that the three-dimensional coordinates of the camera 440 at different positions are known. In various examples, the camera 440 can pivot at a cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., the contents of a blood vessel or vessel) can emit a signal, such as by fluoroscopy, the actual position can be triangulated from the camera 440 being rapidly moved between two or more known positions. In FIG. 10A , the camera 440 is moved axially along axis A. More specifically, the camera 440 is translated, such as by moving in and out with a robotic arm, a distance d1 along axis A closer to the critical structure 401 to a position shown as position 440′. As the camera 440 moves the distance d1 and the size of the view changes relative to the critical structure 401, the distance to the critical structure 401 can be calculated. For example, an axial translation (distance d1) of 4.28 mm may correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, the camera 440 may rotate or sweep along an arc between different positions. Referring now to FIG. 10B, the camera 440 moves axially along axis A and rotates θ3 about axis A. The pivot point 442 about which the camera 440 rotates is located at the cannula / patient interface. In FIG. 10B, the camera 440 translates and rotates to position 440''. As the camera 440 moves and the edge of the view changes relative to the critical structure 401, the distance to the critical structure 401 can be calculated. In FIG. 10B, the distance d2 may be, for example, 9.01 mm, and the angle θ3 may be, for example, 0.9 degrees.

[0054] FIG. 5 illustrates a surgical visualization system 500 that is similar in many respects to surgical visualization system 100. In various examples, surgical visualization system 500 may be a further illustration of surgical visualization system 100. Like surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light at multiple wavelengths, for example, to obtain spectral images of hidden structures. Imaging device 520 may also include a three-dimensional camera and associated electronic processing circuitry, in various examples. Surgical visualization system 500 is shown being utilized during surgery to identify and facilitate avoidance of certain critical structures, such as a ureter 501 a and blood vessels 501 b within an organ 503 (in this example, the uterus) that are not visible on the surface.

[0055] The surgical visualization system 500 uses structured light to measure an emitter-tissue distance d from an emitter 506 on the surgical device 502 to a surface 505 of the uterus 503. e The surgical visualization system 500 is configured to determine the emitter-tissue distance d e Based on this, the device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 is t The surgical visualization system 500 is also configured to extrapolate the tissue-ureter distance d from the ureter 501a to the surface 505. A , and the camera-ureter distance d from the imaging device 520 to the ureter 501a w 1, for example, the surgical visualization system 500 may be configured to determine the distance d, e.g., by spectral imaging and time-of-flight sensors. w In various examples, the surgical visualization system 500 can determine the tissue-ureter distance d based on other distance and / or surface mapping logic described herein. A (i.e., depth) can be determined (e.g., triangulated).

[0056] 11, there is shown a schematic diagram of a control system 600 for a surgical visualization system, such as surgical visualization system 100. Control system 600 is a transformation system that integrates tissue identification by spectral signature and tissue location by structured light to identify critical structures, particularly when these structures are obscured by other tissues, such as fat, connective tissue, blood, and / or other organs. Such techniques may also be useful for detecting tissue anomalies, such as differentiating tumors and / or diseased tissue from healthy tissue within an organ.

[0057] The control system 600 is configured to implement a hyperspectral imaging visualization system that uses molecular responses to detect and identify anatomical structures within a surgical field of view. The control system 600 includes conversion logic 648 for converting tissue data into information usable by the surgeon. For example, wavelength-based variable reflectance against occluding materials can be used to identify critical structures within the anatomy. The control system 600 then combines the identified spectral signatures with structured light data within the image. For example, the control system 600 can be employed to create a three-dimensional data set for surgical applications in a system using augmented image overlay. Techniques employing additional visual information can be employed both intraoperatively and preoperatively. In various examples, the control system 600 is configured to provide a clinician with an alert upon proximity of one or more critical structures. Various algorithms can be employed to guide the surgical procedure and robotic and semi-automated approaches based on proximity to the critical structure(s).

[0058] Projected light arrays are employed to determine tissue shape and motion intraoperatively, or flash lidar may be used for tissue surface mapping.

[0059] The control system 600 is configured to detect the structure(s) of interest, provide an image overlay of the structure(s), measure the distance to the surface of visible tissue, and the distance to the buried / buried structure(s) of interest. In other examples, the control system 600 can measure the distance to the surface of visible tissue or detect the structure(s) of interest and provide an image overlay of the structure(s).

[0060] The control system 600 includes a spectral control circuit 602. The spectral control circuit 602 may be, for example, a field programmable gate array (FPGA) or another suitable circuit configuration, as described herein with reference to FIGS. 2A-2C. The spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing, which may utilize, for example, C / C++ code. The video input processor 606 accepts a video input terminal for control (metadata) data, such as shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608 that includes, for example, a hyperspectral video output terminal for interface control (metadata) data. The video output processor 608 provides the video output signal to an image overlay controller 610.

[0061] The video input processor 606 is coupled to a patient-side camera 612 via a patient isolation circuit 614. As previously mentioned, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include multiple transformers so that the patient is isolated from other circuitry in the system. The camera 612 receives intraoperative images via an optical element 632 and an image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor or any of the image sensor technologies discussed herein in connection with FIG. 2, for example. In one embodiment, the camera 612 outputs images in a 14-bit per pixel signal. It will be understood that higher or lower pixel resolutions may be employed without departing from the scope of the present disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which employs hardware registers 618 and a Nios2 coprocessor 620 to process the camera output signal 613. The color RGB fusion output signal is provided to the video input processor 606 and a laser pulse control circuit 622.

[0062] Laser pulse control circuitry 622 controls laser light engine 624. Laser light engine 624 emits light at multiple wavelengths (λ1, λ2, λ3...λ4) including near infrared (NIR). n ) Laser light engine 624 can operate in multiple modes. In one aspect, laser light engine 624 can operate in, for example, two modes. In a first mode, e.g., a standard operating mode, laser light engine 624 outputs an illumination signal. In a second mode, e.g., a specific mode, laser light engine 624 outputs RGBG light and NIR light. In various examples, laser light engine 624 can operate in a polarization mode.

[0063] Light output 626 from laser light engine 624 illuminates targeted anatomical structures within surgical site 627 during surgery. Laser pulse control circuitry 622 also controls a laser pulse controller 628 for a laser pattern projector 630, which projects a laser light pattern 631, such as a grid or pattern of lines and / or dots, at a predetermined wavelength (λ2) onto the surgical tissue or organ at surgical site 627. Camera 612 receives the output patterned light and reflected light through camera optics 632. Image sensor 634 converts the received light into a digital signal.

[0064] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and to a video input module 636 for reading a laser light pattern 631 projected by a laser pattern projector 630 onto the targeted anatomical structure at the surgical site 627. A processing module 638 processes the laser light pattern 631 and outputs a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing a three-dimensional rendered shape of the tissue or organ of the targeted anatomical structure at the surgical site.

[0065] The first video output signal 640 and the second video output signal 642 contain data representing the location of the structures of interest on the three-dimensional surface model which is provided to an integration module 643. In combination with data from the video output processor 608 of the spectral control circuit 602, the integration module 643 determines the distance d to the buried structures of interest. A (FIG. 1) can be determined (e.g., by triangulation algorithm 644) and the distance d A may be provided to the image overlay controller 610 via a video output processor 646. Such conversion logic may include a conversion logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 located at the surgical site 627.

[0066] Pre-operative data 650 from CT or MRI scans can be employed to register or align certain three-dimensional deformable tissues in various instances. Such pre-operative data 650 can be provided to an integration module 643 and ultimately to an image overlay controller 610 so that such information can be overlaid with the view of the camera 612 and provided to a video monitor 652. Registration of pre-operative data is further described herein and in the aforementioned U.S. patent applications, including, for example, U.S. Patent Application No. 16 / 128,195, filed September 11, 2018, entitled "INTEGRATION OF IMAGING DATA," each of which is incorporated herein by reference in its entirety.

[0067] The video monitor 652 can output the integrated / augmented view from the image overlay controller 610. The clinician can select and / or switch between different views on one or more monitors. On the first monitor 652a, the clinician can switch between (A) a view showing a three-dimensional rendering of the visible tissue and (B) an augmented view in which one or more hidden structures of interest are overlaid on the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can switch, for example, between one or more hidden structures of interest and / or distance measurements to the surface of the visible tissue.

[0068] Control system 600, and / or its various control circuits, may be incorporated into the various surgical visualization systems disclosed herein.

[0069] 12 illustrates a structured (or patterned) light system 700 in accordance with at least one aspect of the present disclosure. As described herein, structured light, for example in the form of stripes or lines, can be projected from a light source and / or projector 706 onto a surface 705 of a targeted anatomical structure to identify the shape and contours of the surface 705. For example, a camera 720, which may be similar in many ways to imaging device 120 (FIG. 1), can be configured to detect the projected light pattern on the surface 705. The deformation of the projected pattern upon impacting the surface 705 allows the vision system to calculate depth and surface information of the targeted anatomical structure.

[0070] In certain instances, invisible (i.e., undetectable) structured light can be used without interfering with other computer vision tasks where the projected pattern may be confused. For example, two exactly opposite repeating patterns of infrared light or very fast frame rates of visible light can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0071] As noted above, the various surgical visualization systems described herein can be utilized to visualize a variety of different types of tissue and / or anatomical structures, including tissue and / or anatomical structures that may be hindered from being visualized by EMR in the visible portion of the spectrum. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissue based on different combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of different constituent materials in the tissue being visualized based on the absorption coefficients of the tissue across various EMR wavelengths. The spectral imaging system can further be configured to characterize the tissue type of the tissue being visualized based on the particular combination of constituent materials. By way of example, FIG. 13A is a graph 2300 illustrating how the absorption coefficients of various biological materials vary across the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 represents the absorption coefficient (e.g., cm) of the biological material. -1 The graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Because different tissue types have different combinations of constituent materials, the tissue type(s) being visualized by the surgical visualization system can be identified and differentiated according to the particular combinations of constituent materials detected. Thus, a spectral imaging system can be configured to emit EMR at many different wavelengths, determine the constituent materials of tissue based on the detected EMR absorption responses at the different wavelengths, and then characterize the tissue type based on the particular detected combination of constituent materials.

[0072] An illustration of the use of spectral imaging techniques to visualize various tissue types and / or anatomical structures is shown in Figure 13B. In Figure 13B, a spectral emitter 2320 (e.g., spectral light source 150) is utilized by an imaging system to visualize a surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from the tissues and / or structures at the surgical site 2325 is received by the image sensor 135 (Figure 2) to visualize the tissues and / or structures, which may be either visible (e.g., located on the surface of the surgical site 2325) or obscured (e.g., underlying other tissues and / or structures at the surgical site 2325). In this example, imaging system 142 (FIG. 2) is able to visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissues that do not conform to known or expected spectral signatures) based on the spectral signatures of various tissue / structure types, each characterized by the different absorption properties (e.g., absorption coefficients) of the constituent materials. The visualized tissues and structures can be displayed on a display screen associated with or coupled to imaging system 142, such as imaging system display 146 (FIG. 2), primary display 2119 (FIG. 18), non-sterile display 2109 (FIG. 18), hub display 2215 (FIG. 19), or device / instrument display 2237 (FIG. 19).

[0073] Additionally, imaging system 142 can be configured to adjust or update the displayed visualization of the surgical site according to the identified tissue and / or structure type. For example, imaging system 142 can display margin 2330a associated with tumor 2332 visualized on a display screen (e.g., display 146). Margin 2330a can indicate the area or amount of tissue that should be resected to ensure complete removal of tumor 2332. Control system 133 ( FIG. 2 ) can be configured to control or update the dimensions of margin 2330a based on the tissues and / or structures identified by imaging system 142. In the illustrated example, imaging system 142 has identified multiple anomalies 2338 within the FOV. Accordingly, control system 133 can adjust displayed margin 2330a to a first updated margin 2330b having dimensions sufficient to encompass the anomalies 2338. The imaging system 142 also identifies an artery 2334 that overlaps the initially displayed margin 2330a (shown by the highlighted area 2336 of the artery 2334). Accordingly, the control system 133 can adjust the displayed margin 2330a to a second updated margin 2330c that has sufficient dimensions to encompass the relevant portion of the artery 2334.

[0074] Additionally, tissues and / or structures can be imaged or characterized according to their reflectance properties across the EMR wavelength spectrum, in addition to, or instead of, the absorption properties described above with respect to FIGS. 13A and 13B. For example, FIGS. 13C-13E show various graphs of the reflectance of different types of tissues or structures across various EMR wavelengths. FIG. 13C is an exemplary graphical representation 1050 of a ureter signature versus obstructions. FIG. 13D is an exemplary graphical representation 1052 of an artery signature versus obstructions. FIG. 13E is an exemplary graphical representation 1054 of a nerve signature versus obstructions. The plots in FIGS. 13C-13E represent the reflectance as a function of wavelength (in nm) of specific structures (ureters, arteries, and nerves) relative to the corresponding reflectances of fat, lung tissue, and blood at the corresponding wavelengths. It should be understood that these graphs are for illustrative purposes only and that other tissues and / or structures may include corresponding detectable reflectance signatures that allow the tissue and / or structure to be identified and visualized.

[0075] In various examples, selected wavelengths for spectral imaging can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site (i.e., "selective spectral" imaging). By utilizing selective spectral imaging, the amount of time required to acquire spectral images can be minimized so that information can be obtained in real time or near real time and utilized intraoperatively. In various examples, wavelengths can be selected by a clinician or by control circuitry based on input by a clinician. In certain examples, wavelengths can be selected based on machine learning and / or big data accessible to the control circuitry, for example, via the cloud.

[0076] The aforementioned application of spectral imaging to tissue can be utilized during surgery to measure distances between a waveform emitter and critical structures shielded by tissue. In one aspect of the present disclosure, and referring now to FIGS. 14 and 15 , a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125 is shown. The time-of-flight sensor system 1104, in certain examples, can be incorporated into the surgical visualization system 100 ( FIG. 1 ). The time-of-flight sensor system 1104 includes a waveform emitter 1106 and a waveform receiver 1108 on the same surgical device 1102. The emitted wave 1124 extends from the emitter 1106 to the critical structure 1101, and the received wave 1125 is reflected from the critical structure 1101 to the receiver 1108. The surgical device 1102 is placed through a trocar 1110 extending into a cavity 1107 of the patient.

[0077] The waveforms 1124, 1125 are configured to penetrate the occluding tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 can be wavelengths in the NIR or SWIR spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or photoacoustic signal can be emitted from the emitter 1106 and can be transmitted through the tissue 1103 obscuring the critical structure 1101. The emitted waveform 1124 can be reflected by the critical structure 1101. The received waveform 1125 can be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various examples, the waveforms 1124, 1125 can be selected to target the critical structure 1101 within the tissue 1103 based on the spectral signature of the critical structure 1101, as described further herein. In various examples, the emitter 1106 is configured to provide a binary signal representing on and off, which can be measured by the receiver 1108, for example as shown in FIG.

[0078] Based on the delay between the emitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d (FIG. 14). A time-of-flight timing diagram 1130 for the emitter 1106 and receiver 1108 of FIG. 14 is shown in FIG. 15. The delay is a function of the distance d, which is given by:

[0079]

number

[0080] As provided herein, the time of flight of waveforms 1124, 1125 corresponds to distance d in Figure 14. In various examples, the additional emitter / receiver and / or pulsed signal from emitter 1106 can be configured to emit a non-penetrating signal. The non-penetrating tissue can be configured to determine the distance from the emitter to the surface 1105 of the shielding tissue 1103. In various examples, the depth of the critical structure 1101 can be determined by the following equation: d A =d w -d t . During the ceremony, d A = depth of critical structure 1101, d w = the distance from the emitter 1106 to the critical structure 1101 (d in FIG. 14), and d t = the distance from the emitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the shielding tissue 1103.

[0081] In one aspect of the disclosure, and referring now to FIG. 16 , a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c is shown. The time-of-flight sensor system 1204, in certain examples, can be incorporated into the surgical visualization system 100 ( FIG. 1 ). The time-of-flight sensor system 1204 includes a waveform emitter 1206 and a waveform receiver 1208. The waveform emitter 1206 is disposed on a first surgical device 1202a, and the waveform receiver 1208 is disposed on a second surgical device 1202b. The surgical devices 1202a, 1202b are disposed through trocars 1210a, 1210b, respectively, extending within a patient's cavity 1207. Emitted waves 1224a, 1224b, 1224c extend from emitter 1206 towards the surgical site, and received waves 1225a, 1225b, 1225c are reflected from receiver 1208 off of various structures and / or surfaces at the surgical site.

[0082] The different emitted waves 1224a, 1224b, 1224c are configured to target different types of material at the surgical site. For example, wave 1224a targets the occluding tissue 1203, wave 1224b targets a first structure of interest 1201a (e.g., a blood vessel), and wave 1224c targets a second structure of interest 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, 1224c can be in the visible, NIR, or SWIR spectrum. For example, visible light can reflect off the surface 1205 of tissue 1203, and the NIR and / or SWIR waveforms can be configured to transmit through the surface 1205 of tissue 1203. In various aspects, spectral signals (e.g., hyperspectral, multispectral, or selective spectrum) or photoacoustic signals can be emitted from emitter 1206 as described herein. In various examples, the waves 1224b, 1224c can be selected to target critical structures 1201a, 1201b within tissue 1203 based on the spectral signatures of the critical structures 1201a, 1201b, as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, each of which is incorporated by reference in its entirety into this disclosure.

[0083] The emitted waves 1224a, 1224b, 1224c may be reflected from the targeted material (i.e., surface 1205, first structure of interest 1201a, and second structure of interest 1201b, respectively). The received waveforms 1225a, 1225b, 1225c are reflected from the target material (i.e., surface 1205, first structure of interest 1201a, and second structure of interest 1201b, respectively) at a distance d shown in FIG. 1a , d 2a , d 3a , d 1b , d 2b , d 2c There may be delays due to:

[0084] In a time-of-flight sensor system 1204 in which the emitter 1206 and receiver 1208 can be independently positioned (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d 1a , d 2a , d 3a, d 1b , d 2b , d 2c can be calculated from the known positions of the emitter 1206 and receiver 1208. For example, the positions may be known when the surgical devices 1202a, 1202b are robotically controlled. With knowledge of the positions of the emitter 1206 and receiver 1208, as well as the time of the photon stream targeting a particular tissue and its specific response information received by the receiver 1208, the distance d 1a , d 2a , d 3a , d 1b , d 2b , d 2c In one aspect, the distance to the occluded critical structures 1201 a, 1201 b can be triangulated using the transmitted wavelengths. Because the speed of light is constant for any wavelength of light, visible or invisible, the time-of-flight sensor system 1204 can determine a variety of distances.

[0085] With further reference to FIG. 16 , in various examples, the view provided to the clinician can rotate the receiver 1208 so that the center of mass of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such an orientation can quickly communicate one or more relevant distances and / or perspectives relative to the critical structures. For example, as shown in FIG. 16 , the surgical site is displayed from a perspective in which the critical structure 1201a is perpendicular to the view plane (i.e., the vessels are pointing into or out of the page). In various examples, such an orientation can be a default setting. However, the view can be rotated or otherwise adjusted by the clinician. In certain examples, the clinician can switch between different surfaces and / or target structures that define the perspective of the surgical site provided by the imaging system.

[0086] In various examples, the receiver 1208 can be mounted on a trocar or cannula, such as trocar 1210b, through which the surgical device 1202b is placed. In other examples, the receiver 1208 can be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 1208 can be mounted on a movable arm separate from the robot controlling the surgical device 1202a, or can be mounted on an operating room (OR) table that can be aligned to the robot coordinate plane during surgery. In such examples, the positions of the emitter 1206 and the receiver 1208 can be aligned to the same coordinate plane so that distances can be triangulated from the output of the time-of-flight sensor system 1204.

[0087] The combination of a time-of-flight sensor system with near-infrared spectroscopy (NIRS), termed TOF-NIRS, which is capable of measuring time-resolved profiles of NIR light with nanosecond resolution, can be found in the article entitled "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT," Journal of the American Society for Horticultural Science, May 2013, vol. 138, no. 3, pp. 225-228, which is incorporated herein by reference in its entirety and can be viewed at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0088] In various examples, the time-of-flight spectral waveform is configured to determine the depth of a critical structure and / or the proximity of a surgical device to the critical structure. Additionally, various surgical visualization systems disclosed herein include surface mapping logic configured to create a three-dimensional rendering of the surface of the visible tissue. In such examples, even if the critical structure is obscured by the visible tissue, the clinician can recognize the proximity (or lack thereof) of the surgical device to the critical structure. In one example, the surface mapping logic provides a topography of the surgical site on a monitor. If the critical structure is near the surface of the tissue, spectral imaging can communicate the location of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In another example, spectral imaging can detect structures 10 or 20 mm below the surface of the tissue. Based on known limitations of spectral imaging systems, the system is configured to communicate that a critical structure is out of range if it is simply not detected by the spectral imaging system. Thus, the clinician can continue to move the surgical device and / or manipulate the tissue. When a structure of interest moves within range of the spectral imaging system, the system can identify the structure and therefore communicate that the structure is within range. In such an example, an alert can be provided when the structure is first identified and / or further moved within a pre-defined proximity zone. In such an example, proximity information (i.e., not being in proximity) can be provided to the clinician even if the structure of interest is not identified by the spectral imaging system with known boundaries / ranges.

[0089] The various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structure(s) during surgery and alert the clinician before inadvertent dissection and / or transection damages the critical structure(s). In various aspects, the surgical visualization system is configured to identify one or more of the following critical structures: ureter, intestine, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], facial nerve process, vagus nerve, and their branches), blood vessels (including the pulmonary and lobar arteries and veins, the inferior mesenteric artery [IMA] and its branches, the superior rectal artery, the sigmoid artery, and the left colic artery), the superior mesenteric artery (SMA) and its branches (including the middle colic artery, the right colic artery, and the ileocolic artery), the hepatic artery and its branches, the portal vein and its branches, the splenic artery / vein and their branches, the external and internal iliac vessels (lower abdominal), the short gastric artery, the uterine artery, the median sacral vessels, and lymph nodes. Additionally, the surgical visualization system is configured to indicate the proximity of the surgical device(s) to the critical structure(s) and / or alert the clinician when the surgical device(s) approach the critical structure.

[0090] Various aspects of the present disclosure provide for intraoperative identification of critical structures (e.g., ureter, nerve, and / or blood vessel identification) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein can include spectral imaging and surgical instrument tracking, enabling visualization of critical structures below the tissue surface, e.g., 1.0-1.5 cm below the tissue surface. In other examples, a surgical visualization system can identify structures less than 1.0 cm or more than 1.5 cm below the tissue surface. For example, a surgical visualization system that can only identify structures within 0.2 mm of the surface can be useful, for example, when they are otherwise invisible due to depth. In various aspects, a surgical visualization system can augment the clinician's view by, for example, virtually displaying critical structures as a visible white-light image overlay on the visible tissue surface. A surgical visualization system can provide real-time three-dimensional spatial tracking of the distal tip of a surgical instrument and can provide a proximity alert when the distal tip of a surgical instrument moves within a specified range of a critical structure, e.g., within 1.0 cm of the critical structure.

[0091] Various surgical visualization systems disclosed herein can identify when an incision is too close to a critical structure. An incision can be “too close” to a critical structure based on temperature (i.e., too high a temperature near the critical structure, which may pose a risk of damaging / heating / melting the critical structure) and / or tension (i.e., too much tension near the critical structure, which may pose a risk of damaging / tearing / pulling the critical structure). Such a surgical visualization system can facilitate perivascular incisions, for example, when skeletonizing perivascular tissue prior to ligation. In various examples, an infrared camera can be utilized to read heat at the surgical site and provide a warning to the clinician based on the detected heat and the distance from the instrument to the structure. For example, if the instrument temperature exceeds a predetermined threshold (e.g., 120°F), an alert can be provided to the clinician at a first distance (e.g., 10 mm), and if the instrument temperature is at or below the predetermined threshold, an alert can be provided to the clinician at a second distance (e.g., 5 mm). The pre-defined thresholds and / or warning distances may be default settings and / or clinician programmable. Additionally or alternatively, proximity alerts may be linked to thermal measurements made on the instrument itself, such as, for example, a thermocouple measuring heat within the distal jaws of a monopolar or bipolar cutting instrument or vessel sealer.

[0092] The various surgical visualization systems disclosed herein can provide sufficient sensitivity and specificity for critical structures to enable clinicians to confidently proceed with rapid but safe dissections based on standards of care and / or device safety data. The systems can function in real time during a surgical procedure with minimal, and in various instances, no, ionizing radiation risk to the patient or clinician. Conversely, in fluoroscopy procedures, the patient and clinician(s) may be exposed to ionizing radiation, for example, via x-ray beams utilized to view anatomical structures in real time.

[0093] The various surgical visualization systems disclosed herein can be configured to detect and identify one or more desired types of critical structures in the forward path of a surgical device, such as when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system can be configured to detect and identify one or more types of critical structures in the area surrounding the surgical device and / or in multiple planes / dimensions, for example.

[0094] The various surgical visualization systems disclosed herein are easy to operate and / or interpret. Additionally, various surgical visualization systems may incorporate an "override" feature that allows a clinician to override default settings and / or operations. For example, a clinician may selectively turn off alerts from the surgical visualization system and / or may approach a critical structure closer than suggested by the surgical visualization system, such as when the risk to a critical structure is lower than the risk of avoiding that area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue behind may be greater than the risk of damaging the critical structure).

[0095] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, implementation of the surgical visualization system may not change the way the surgical procedure is performed. Furthermore, the surgical visualization system may be economical compared to the costs of inadvertent transections. Data has shown a reduction in inadvertent damage to critical structures, which may facilitate increased reimbursement.

[0096] The various surgical visualization systems disclosed herein can operate in real-time or near real-time and sufficiently in advance to allow the clinician to predict critical structure(s). For example, the surgical visualization system can provide sufficient time for "slow down, assess, and avoid" to maximize the efficiency of the surgical procedure.

[0097] Various surgical visualization systems disclosed herein may not require contrast agents or dyes to be injected into tissue. For example, spectral imaging is configured to visualize hidden structures during surgery without the use of contrast agents or dyes. In other examples, contrast agents may be easier to inject into the appropriate layer(s) of tissue than with other visualization systems. The time between injection of contrast agents and visualization of critical structures may be less than two hours, for example.

[0098] The various surgical visualization systems disclosed herein can be linked with clinical and / or equipment data. For example, the data can provide boundaries regarding how close an energized surgical device (or other potentially damaging device) can be from tissue the surgeon does not want to damage. Any data modules that interface with the surgical visualization systems disclosed herein can be provided integrally with or separately from a robot, for example, to enable use with standalone surgical devices in open or laparoscopic procedures. The surgical visualization system, in various examples, can be compatible with robotic surgical systems. For example, visualization images / information can be displayed within the robotic console.

[0099] In various instances, a clinician may be unsure of the location of a critical structure relative to a surgical tool. For example, if the critical structure is embedded in tissue, the clinician may be unable to locate the critical structure. In certain instances, the clinician may desire to maintain the surgical device out of range of locations surrounding the critical structure and / or away from visible tissue covering the hidden critical structure. If the location of the hidden critical structure is unknown, the clinician risks approaching the critical structure too closely, potentially resulting in inadvertent trauma and / or dissection of the critical structure and / or applying excessive energy, heat, and / or tension near the critical structure. Alternatively, the clinician may remain too far away from the suspected location of the critical structure, potentially affecting tissue in less desirable locations in an effort to avoid the critical structure.

[0100] A surgical visualization system is provided that provides tracking of a surgical device relative to one or more critical structures. For example, the surgical visualization system can track the proximity of the surgical device to critical structures. Such tracking can occur in real time and / or near real time during surgery. In various examples, the tracking data can be provided to a clinician via a display screen (e.g., a monitor) of the imaging system.

[0101] In one aspect of the present disclosure, a surgical visualization system includes a surgical device having an emitter configured to emit a structured light pattern onto a visible surface; an imaging system including a camera configured to detect the structured light pattern on the implanted structure and the visible surface; and control circuitry in signal communication with the camera and the imaging system, the control circuitry configured to determine a distance from the surgical device to the implanted structure and provide a signal indicative of the distance to the imaging system. For example, the distance can be determined based on a three-dimensional view of the illuminated structure provided by images from multiple lenses (e.g., left and right lenses) of the camera by calculating the distance from the camera to the fluoroscopically illuminated critical structure. The distance from the surgical device to the critical structure can be triangulated, for example, based on the known positions of the surgical device and the camera. Alternative means for determining the distance to the implanted critical structure are further described herein. For example, an NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue that overlaps / covers the implanted critical structure. For example, the surgical visualization system can identify hidden structures of interest and enhance the view of the hidden structures of interest by outlining the hidden structures of interest on the visible structures, such as lines on the surface of the visible tissue, and can further determine the distance to the enhanced lines on the visible tissue.

[0102] By providing clinicians with up-to-date information regarding the proximity of a surgical device to hidden critical structures and / or visible structures, as provided by the various surgical visualization systems disclosed herein, clinicians can make more informed decisions regarding the placement of a surgical device relative to hidden critical structures. For example, clinicians can view the distance between the surgical device and critical structures in real time / during surgery, and in certain instances, alerts and / or warnings can be provided by the imaging system when the surgical device is moved within a predetermined proximity and / or zone of a critical structure. In certain instances, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a "no-fly" zone near a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm or more of a critical structure). In such instances, clinicians can maintain momentum throughout a surgical procedure without requiring clinician monitoring of suspected locations of critical structures and the proximity of the surgical device to these locations. As a result, certain surgical procedures can be performed more quickly, with fewer pauses / interruptions, and / or with improved precision and / or reliability. In one aspect, a surgical visualization system can be utilized to detect tissue variations, such as variations in tissue within an organ, to differentiate tumor / cancerous / pathological tissue from healthy tissue. Such a surgical visualization system can maximize the removal of pathological tissue while minimizing the removal of healthy tissue.

[0103] Surgical Hub System The various visualization or imaging systems described herein can be incorporated into a surgical hub system such as that illustrated in connection with Figures 17-19 and described in further detail below.

[0104] 17 , a computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104 that may include the remote server 2113. In one example, as shown in FIG. 17 , a surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112, which are configured to communicate with each other and / or with the hub 2106. In some embodiments, a surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to 1.

[0105] FIG. 18 shows an example of a surgical system 2102 being used to perform a surgical procedure on a patient lying on an operating table 2114 in an operating room 2116. A robotic system 2110 is used as part of the surgical system 2102 in the surgical procedure. The robotic system 2110 includes a surgeon's console 2118, a patient side cart 2120 (surgical robot), and a surgical robot hub 2122. The patient side cart 2120 can manipulate at least one detachably coupled surgical tool 2117 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 2118. Images of the surgical site are acquired by a medical imaging device 2124, which can be manipulated by the patient side cart 2120 to orient the imaging device 2124. The robotic hub 2122 can be used to process and then display the images of the surgical site to the surgeon through the surgeon's console 2118.

[0106] Other types of robotic systems can be readily adapted for use with surgical system 2102. Various examples of robotic system surgical tools suitable for use with the present disclosure are described in various U.S. patent applications incorporated herein by reference in this disclosure.

[0107] Various examples of cloud-based analytics performed by the cloud 2104 and suitable for use with the present disclosure are described in various US patent applications incorporated herein by reference in this disclosure.

[0108] In various embodiments, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors.

[0109] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0110] The one or more illumination sources may be configured to emit electromagnetic energy within the visible spectrum as well as the invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or luminous spectrum, is the portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and is sometimes referred to as visible light or simply light. The typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0111] The invisible spectrum (i.e., non-radiative spectrum) is the portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which constitutes invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which constitutes invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0112] In various aspects, the imaging device 2124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngological-nephroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0113] In one aspect, the imaging device employs multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within specific wavelength ranges across the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, e.g., IR and UV light. Spectral imaging can extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in various U.S. patent applications incorporated herein by reference in this disclosure. Multispectral monitoring can be a useful tool for repositioning the surgical field after the completion of a surgical task to perform one or more of the above-mentioned tests on the treated tissue.

[0114] It is self-evident that any surgical procedure requires rigorous sterilization of the operating room and surgical equipment. The strict hygiene and sterilization conditions required in the "surgical field," i.e., the operating room or procedure room, necessitate the highest possible sterility of all medical device equipment and instruments. Part of the sterilization process described above includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 2124 and its accessories and components. It is understood that the sterile field may be considered a specific area deemed free of microorganisms, such as in a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is prepared for surgery. The sterile field may include properly attired, cleaned team members, and all supplies and fixtures within the area. In various aspects, the visualization system 2108 includes one or more imaging sensors strategically positioned relative to the sterile field, as shown in FIG. 18 , one or more image processing units, one or more storage arrays, and one or more displays. In one aspect, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. The various components of the visualization system 2108 are described in various US patent applications which are incorporated herein by reference in this disclosure.

[0115] As shown in FIG. 18 , primary display 2119 is positioned in the sterile field so as to be visible to the operator of operating table 2114. In addition, visualization tower 21121 is positioned outside the sterile field. Visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 facing opposite each other. Visualization system 2108, guided by hub 2106, is configured to utilize displays 2107, 2109, and 2119 to coordinate information flow to operators inside and outside the sterile field. For example, hub 2106 can cause visualization system 2108 to display snapshots of the surgical site captured by imaging device 2124 on non-sterile display 2107 or 2109 while maintaining a live video of the surgical site on primary display 2119. The snapshots on non-sterile display 2107 or 2109 can, for example, enable a non-sterile operator to perform diagnostic procedures related to the surgical procedure.

[0116] In one aspect, the hub 2106 is also configured to send diagnostic input or feedback entered by the non-sterile operator at the visualization tower 21121 to the primary display 2119 in the sterile field for viewing by the sterile operator at the operating table. In one example, the input can be in the form of modifications to a snapshot displayed on the non-sterile display 2107 or 2109, which can be sent by the hub 2106 to the primary display 2119.

[0117] 18 , a surgical instrument 2112 is used as part of a surgical system 2102 in a surgical procedure. The hub 2106 is also configured to coordinate information flow to the display of the surgical instrument 2112, as described in various U.S. patent applications incorporated by reference in this disclosure. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 can be sent by the hub 2106 to the surgical instrument display 2115 in the sterile field, where it can be viewed by the operator of the surgical instrument 2112. Exemplary surgical instruments suitable for use with the surgical system 2102 are described in various U.S. patent applications incorporated by reference in this disclosure.

[0118] 19 illustrates a computer-implemented interactive surgical system 2200. The computer-implemented interactive surgical system 2200 is similar in many respects to the computer-implemented interactive surgical system 2100. Each surgical system 2200 includes at least one surgical hub 2236 in communication with a cloud 2204, which may include a remote server 2213. In one aspect, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to multiple surgical field devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. 19 , the surgical hub 2236 is coupled to an imaging module 2238 coupled to an endoscope 2239, a generator module 2240 coupled to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communications module 2230, a processor module 2232, a storage array 2234, a smart device / instrument 2235 coupled to an optional display 2237, and a non-contact sensor module 2242. The surgical field devices are coupled to cloud computing resources and data storage via the surgical hub 2236. The robotic hub 2222 may also be connected to the surgical hub 2236 and cloud computing resources. As described herein, the devices / instruments 2235, visualization system 2209, among others, can be coupled to the surgical hub 2236 via wired or wireless communication standards or protocols. The surgical hub 2236 can be coupled to a hub display 2215 (e.g., monitor, screen) for displaying and overlaying images received from the imaging modules, device / instrument displays, and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower along with the images and overlayed images.

[0119] Situational Awareness The various visualization systems, or aspects of visualization systems, described herein can be utilized as part of a situational awareness system that may be embodied or executed by the surgical hubs 2106, 2236 (FIGS. 17-19). Specifically, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their positions, orientations, and actions), tissues, structures, users, and other objects located within the surgical field or operating room can provide contextual data that can be utilized by the situational awareness system to infer the type of surgical procedure or step being performed, the type of tissue(s) and / or structure(s) being manipulated by the surgeon, etc. This contextual data can then be utilized by the situational awareness system to provide alerts to the user, suggest subsequent steps or actions for the user to take, anticipate preparation of surgical device use (e.g., activating an electrosurgical generator in anticipation of an electrosurgical instrument being used in a later step of a surgical procedure), intelligent control of surgical instruments (e.g., customizing operating parameters of a surgical instrument based on each patient's particular health profile), etc.

[0120] While “intelligent” devices that include control algorithms responsive to sensed data may offer an improvement over “dumb” devices that operate without considering the sensed data, some sensed data may be incomplete or inconclusive when considered alone, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowledge of the surgical context (e.g., without knowledge of the type of tissue being operated on or the type of procedure being performed), a control algorithm may inaccurately or suboptimally control a modular device with particular sensed data without context. Modular devices can include any surgical device controllable with a situational awareness system, such as a visualization system device (e.g., a camera or display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a smoke evacuator). For example, the optimal manner of a control algorithm for controlling a surgical instrument in response to a particular sensed parameter may vary depending on the particular type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., resistance to tearing) and therefore respond differently to actions taken by the surgical instrument. Thus, even when the same measurement value for a particular parameter is detected, it may be desirable for the surgical instrument to take different actions. As one specific example, the optimal manner in which a surgical stapling and cutting instrument controls in response to the instrument detecting an unexpectedly high force to close its end effector differs depending on whether the tissue type is susceptible to tearing or resistant to tearing. For tissue that is susceptible to tearing, such as lung tissue, the instrument's control algorithm optimally decelerates the motor in response to the unexpectedly high force to close to avoid tearing the tissue. For tissue that is resistant to tearing, such as stomach tissue, the instrument's control algorithm optimally accelerates the motor in response to the unexpectedly high force to close to ensure that the end effector is properly clamped to the tissue.Without knowing whether lung tissue or stomach tissue is being clamped, the control algorithm may make suboptimal decisions.

[0121] One solution utilizes a surgical hub including a system configured to derive information about the surgical procedure being performed based on data received from various data sources and then control paired modular devices accordingly. In other words, the surgical hub is configured to infer information about the surgical procedure from the received data and then control the modular devices paired with the surgical hub based on the inferred context for the surgical procedure. FIG. 20 shows a diagram of a context-aware surgical system 2400 in accordance with at least one aspect of the present disclosure. In some examples, data sources 2426 include, for example, modular devices 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular devices themselves), databases 2422 (e.g., EMR databases containing patient records), and patient monitoring devices 2424 (e.g., blood pressure (BP) monitors and electrocardiogram (EKG) monitors).

[0122] The surgical hub 2404 may be similar in many respects to the hub 106 and may be configured to derive contextual information regarding a surgical procedure from the data, for example, based on a particular combination or combinations of received data or the particular order in which the data is received from the data sources 2426. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure the surgeon is performing, the type of tissue being operated on, or the body cavity being treated. This ability of some aspects of the surgical hub 2404 to derive or infer information regarding a surgical procedure from received data may be referred to as “situational awareness.” In one example, the surgical hub 2404 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 2404 that derives contextual information related to a surgical procedure from received data.

[0123] The situational awareness system of the surgical hub 2404 can be configured to derive contextual information from data received from the data sources 2426 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained with training data to correlate various inputs (e.g., data from the database 2422, the patient monitoring devices 2424, and / or the modular devices 2402) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from provided inputs. In another example, the situational awareness system can include a lookup table that stores pre-characterized contextual information about the surgical procedure in association with one or more inputs (or ranges of inputs) corresponding to the contextual information. In response to querying with one or more inputs, the lookup table can return corresponding contextual information for the situational awareness system to control the modular devices 2402. In one example, the contextual information received by the surgical hub 2404 situational awareness system is associated with a particular control adjustment or set of control adjustments for one or more modular devices 2402. In another example, the situational awareness system includes an additional machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 2402 when provided with the contextual information as input.

[0124] A surgical hub 2404 incorporating a situational awareness system provides many benefits to the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which improves the accuracy of processing and / or use of the data during the course of a surgical procedure. Returning to the previous example, the situational aware surgical hub 2404 can determine what type of tissue is being operated on, and thus, if an unexpectedly high force to close the end effector of a surgical instrument is detected, the situational aware surgical hub 2404 can properly accelerate or decelerate the motor of the surgical instrument to match the tissue type.

[0125] As another example, the type of tissue being operated on may affect the adjustments made to the compression speed and load threshold of a surgical stapling and cutting instrument for a particular tissue gap measurement. The context-aware surgical hub 2404 may infer whether the surgical procedure being performed is thoracic or abdominal surgery, which allows the surgical hub 2404 to determine whether the tissue being clamped by the end effector of the surgical stapling and cutting instrument is pulmonary (in the case of thoracic surgery) or stomach (in the case of abdominal surgery). The surgical hub 2404 may then adjust the compression speed and load threshold of the surgical stapling and cutting instrument appropriately for the tissue type.

[0126] As yet another example, the type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situation-aware surgical hub 2404 can determine if the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the procedure type. Since certain procedure types are generally performed within specific body cavities, the surgical hub 2404 can control the motor speed of the smoke evacuator appropriately for the body cavity being operated on. Thus, the situation-aware surgical hub 2404 can provide a consistent amount of smoke evacuation for both thoracic and abdominal procedures.

[0127] As yet another implementation example, the type of procedure being performed can affect the optimal energy level at which an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument operates. For example, an arthroscopic procedure requires a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The context-aware surgical hub 2404 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 can then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on can affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The context-aware surgical hub 2404 can determine which type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, according to the tissue geometry expected for the surgery. Furthermore, the context-aware surgical hub 2404 can be configured to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument over the course of the surgical procedure, rather than just on a procedure-by-procedure basis. The situation-aware surgical hub 2404 can determine which step of the surgical procedure is being performed or will continue to be performed and then update the generator and / or control algorithms of the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level to a value appropriate for the expected tissue type according to the step of the surgical procedure.

[0128] As yet another example, the surgical hub 2404 may derive data from additional data sources 2426 to improve conclusions drawn from one data source 2426. The context-aware surgical hub 2404 may augment the data received from the modular device 2402 with contextual information constructed about the surgical procedure from other data sources 2426. For example, the context-aware surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, the video or image data may be inconclusive. Thus, in one example, the surgical hub 2404 may be further configured to compare a physiological measurement (e.g., blood pressure sensed by a BP monitor communicatively connected to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 ( FIG. 2 ) communicatively coupled to the surgical hub 2404) to make a determination regarding the integrity of a staple line or tissue weld. In other words, the surgical hub's 2404 situational awareness system can take physiological measurement data into account to provide additional context when analyzing the visualization data, which can be useful when the visualization data may not be conclusive or incomplete on its own.

[0129] Another advantage includes actively and automatically controlling paired modular devices 2402 according to the particular step of the surgical procedure being performed to reduce the number of times a medical professional is required to interact with or control the surgical system 2400 during the course of a surgical procedure. For example, the context-aware surgical hub 2404 can actively activate a generator to which an RF electrosurgical instrument is connected if it is determined that a subsequent step in the procedure will require the use of the instrument. By actively activating the energy source, the instrument can be ready for use as soon as the previous step in the procedure is completed.

[0130] As another example, the context-aware surgical hub 2404 can determine whether a current or subsequent step in a surgical procedure requires a different view or magnification on the display according to the feature(s) of the surgical site that the surgeon is expected to need to see. The surgical hub 2404 can then proactively change the displayed view (e.g., provided by a medical imaging device for the visualization system 108) appropriately, causing the display to automatically adjust throughout the surgical procedure.

[0131] As yet another example, the context-aware surgical hub 2404 can determine which step of the surgical procedure is being performed or will be performed next, and whether specific data or data comparisons are required for that step of the surgical procedure. The surgical hub 2404 can be configured to automatically call up data screens based on the step of the surgical procedure being performed, without waiting for the surgeon to request specific information.

[0132] Another benefit includes checking for errors during surgical setup or during the course of a surgical procedure. For example, the situation-aware surgical hub 2404 can determine whether the surgical field is properly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve (e.g., from memory) the corresponding checklist, product locations, or setup requirements, and then compare the current surgical field layout to a standard layout for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 can be configured to compare, for example, a list of items for the procedure scanned with a preferred scanner and / or a list of devices paired with the surgical hub 2404 with a recommended or predicted inventory of items and / or devices for a given surgical procedure. If a discontinuity exists between the lists, the surgical hub 2404 can be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing. In one example, the surgical hub 2404 can be configured to determine the relative distance or relative position of the modular device 2402 and the patient monitoring device 2424, for example, via proximity sensors. The surgical hub 2404 can compare the relative positions of the devices to a recommended or predicted layout for a particular surgical procedure. If a discontinuity exists between the layouts, the surgical hub 2404 can be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0133] As another example, the context-aware surgical hub 2404 can determine whether a surgeon (or other medical personnel) is making an error or deviating from an expected sequence of actions during the course of a surgical procedure. For example, the surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve (e.g., from memory) a corresponding list of steps or sequences of equipment use, and then compare the steps being performed or the equipment being used during the surgical procedure with the expected steps or equipment for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 can be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0134] Overall, the situational awareness system for the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure (e.g., adjusting for different tissue types) and validating actions during surgery. The situational awareness system also improves the surgeon's efficiency in performing surgical procedures by automatically suggesting next steps, providing data, and adjusting displays and other modular devices 2402 within the surgical field according to the specific context of the procedure.

[0135] 21, a timeline 2500 illustrating the situational awareness of a hub, such as the surgical hub 106 or 206 (FIGS. 1-11), is shown. The timeline 2500 illustrates an exemplary surgical procedure and the contextual information that the surgical hub 106, 206 can derive from data received from data sources at each step of the surgical procedure. The timeline 2500 illustrates the general steps that nurses, surgeons, and other medical personnel may take during the course of a lung segmentectomy surgery, beginning with the setup of the surgical site and concluding with the transfer of the patient to the post-operative recovery room.

[0136] The context-aware surgical hub 106, 206 receives data from data sources throughout the course of a surgical procedure, including data generated each time a medical professional utilizes a modular device paired with the surgical hub 106, 206. The surgical hub 106, 206 receives this data from the paired modular devices and other data sources and can continuously derive inferences (i.e., contextual information) about the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The context-aware system of the surgical hub 106, 206 can, for example, record data about the procedure to generate a report, verify the step the medical professional is performing, provide data or prompts (e.g., via a display screen) that may be relevant to a particular surgical step, adjust modular devices based on the context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument, etc.), and any other of these actions described above.

[0137] As a first step 2502 in this exemplary procedure, hospital personnel retrieve the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hub 106, 206 determines that the procedure to be performed is thoracic surgery.

[0138] In a second step 2504, personnel scan the medical supplies arriving for the procedure. The surgical hub 106, 206 cross-references the scanned supplies with a list of supplies utilized in various types of procedures to verify that the combination of supplies matches a thoracic procedure. Additionally, the surgical hub 106, 206 may also determine that the procedure is not a wedge procedure (either because the arriving supplies do not include specific supplies required for a thoracic wedge procedure or because they are otherwise not compatible with a thoracic wedge procedure).

[0139] In a third step 2506, the medical personnel scans the patient band via a scanner communicatively connected to the surgical hub 106, 206. The surgical hub 106, 206 can then verify the patient's identity based on the scanned data.

[0140] In a fourth step 2508, the medical staff turns on the auxiliary equipment. The auxiliary equipment utilized may vary depending on the type of surgical procedure and the technology used by the surgeon, but in this exemplary case includes a smoke evacuator, an insufflator, and a medical imaging device. When the auxiliary equipment is activated, the modular device may automatically pair with the surgical hub 106, 206 located within a specific proximity of the modular device as part of its initialization process. The surgical hub 106, 206 can then derive contextual information about the surgical procedure by detecting the type of modular device that is paired during this pre-operative or initialization phase. In this particular example, the surgical hub 106, 206 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices. Based on a combination of data from the patient's EMR, a list of medical supplies to be used in the procedure, and the type of modular devices connecting to the hub, the surgical hub 106, 206 can generally infer the specific procedure the surgical team will perform. Once the surgical hub 106, 206 knows which particular procedure is being performed, it can then retrieve the steps of that procedure from memory or from the cloud and then cross-reference data it subsequently receives from connected data sources (e.g., modular devices and patient monitoring devices) to infer which steps of the surgical procedure the surgical team is performing.

[0141] In a fifth step 2510, personnel attach EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices may be paired with the surgical hub 106, 206. When the surgical hub 106, 206 begins receiving data from the patient monitoring devices, the surgical hub 106, 206 confirms that the patient is present at the surgical site.

[0142] In a sixth step 2512, medical personnel anesthetize the patient. The surgical hub 106, 206 can infer that the patient is under anesthesia based on data from modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 2512 is complete, the pre-operative portion of the lung segmentectomy surgery is complete and the surgical portion begins.

[0143] A seventh step 2514 is to collapse the patient's lung being operated on (while switching ventilation to the contralateral lung). The Surgical Hub 106, 206 can, for example, infer from ventilator data that the patient's lung has been collapsed. Because the Surgical Hub 106, 206 can compare the detection of the patient's lung being collapsed to the expected steps of the procedure (which may be accessed or retrieved in advance), it can infer that the surgical portion of the procedure has begun and thereby determine that collapsing the lung is the first surgical step in this particular procedure.

[0144] In an eighth step 2516, a medical imaging device (e.g., a scope) is inserted and video capture from the medical imaging device begins. The Surgical Hub 106, 206 receives the medical imaging device data (i.e., video or image data) through the connection to the medical imaging device. Upon receiving the medical imaging device data, the Surgical Hub 106, 206 can determine that the laparoscopic portion of the surgical procedure has begun. Furthermore, the Surgical Hub 106, 206 can determine that the particular procedure being performed is a segmentectomy, rather than a lobectomy (note that a wedge procedure was already determined to be ruled out by the Surgical Hub 106, 206 based on the data received in the second step 2504 of the procedure). Data from the medical imaging device 124 (FIG. 2) can be used to determine contextual information about the type of procedure being performed in several different ways, including determining the angle at which the medical imaging device is oriented with respect to visualization of the patient's anatomy, monitoring the number of medical imaging devices being utilized (i.e., activated and paired with the surgical hub 106, 206), and monitoring the type of visualization device being utilized. For example, one technique for performing a VATS lobectomy positions the camera above the diaphragm in the anterior-inferior corner of the patient's chest cavity, while one technique for performing a VATS segmentectomy positions the camera in an intercostal position anterior to the segmental fissure. The situational awareness system can be trained to recognize the position of the medical imaging device according to the visualization of the patient's anatomy, for example, using pattern recognition or machine learning techniques. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively linked to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not utilized in a VATS lobectomy.By tracking any or all of this data from the medical imaging devices, the surgical hub 106, 206 can determine the particular type of surgical procedure being performed and / or the techniques being used for the particular type of surgical procedure.

[0145] In a ninth step 2518, the surgical team begins the incision step of the procedure. Because the surgical hub 106, 206 receives data from the RF or ultrasonic generator indicating that an energy instrument is being fired, it can infer that the surgeon is in the process of incising and moving the patient's lungs. The surgical hub 106, 206 can cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy instrument being fired at this point in the process (i.e., after the steps of the procedure described above have been completed) corresponds to the incision step. In a particular example, the energy instrument may be an energy instrument mounted on a robotic arm of a robotic surgical system.

[0146] In a tenth step 2520, the surgical team proceeds to the ligation step of the procedure. Because the surgical hub 106, 206 receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired, it can infer that the surgeon is currently ligating an artery and a vein. As with the previous step, the surgical hub 106, 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the retrieved step of the process. In a particular example, the surgical instrument may be a surgical tool mounted on a robotic arm of a robotic surgical system.

[0147] In an eleventh step 2522, the segmental resection portion of the procedure is performed. The surgical hub 106, 206 can infer that the surgeon is transecting parenchyma based on data from the surgical stapling and cutting instrument, including data from its cartridge. The cartridge data can correspond, for example, to the size or type of staples being fired by the instrument. Because different types of staples are used on different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transected. In this case, the type of staples being fired is used on the parenchyma (or other similar tissue type), thereby enabling the surgical hub 106, 206 to infer that the segmental resection portion of the procedure is being performed.

[0148] Subsequently, in a twelfth step 2524, a node dissection step is performed. The surgical hub 106, 206 can infer that the surgical team is dissecting nodes and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. For this particular procedure, the RF or ultrasonic instrument utilized after the parenchymal tissue is transected corresponds to the node dissection step, thereby enabling the surgical hub 106, 206 to make the above inference. Note that surgeons regularly alternate between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the particular step in the procedure, as different instruments are better suited for specific tasks. Thus, the particular sequence in which the stapling / cutting instrument and the surgical energy instrument are used can indicate which step of the procedure the surgeon is performing. Furthermore, in certain instances, robotic tools may be utilized for one or more steps in the surgical procedure, and / or handheld surgical instruments may be utilized for one or more steps in the surgical procedure. The surgeon(s) can, for example, alternate between using robotic tools and handheld surgical instruments and / or use the devices simultaneously. Once the twelfth step 2524 is completed, the incisions are closed and the post-operative portion of the procedure begins.

[0149] In a thirteenth step 2526, the patient is awakened from anesthesia. The surgical hub 106, 206 may infer that the patient is awakening from anesthesia, for example, based on ventilator data (i.e., the patient's breathing rate begins to increase).

[0150] Finally, a fourteenth step 2528 is for medical personnel to remove the various patient monitoring devices from the patient. Thus, the surgical hub 2106, 2236 can infer that the patient is being transferred to a recovery room when the hub loses EKG data, BP data, and other data from the patient monitoring devices. As can be seen from the description of this exemplary procedure, the surgical hub 2106, 2236 can determine or infer when each step of a given surgical procedure is occurring according to data received from various data sources communicatively coupled to the surgical hub 2106, 2236.

[0151] Situational awareness is further described in various U.S. patent applications incorporated herein by reference in their entireties in this disclosure. In certain examples, the operation of a robotic surgical system, including, for example, the various robotic surgical systems disclosed herein, can be controlled by the hub 2106, 2236 based on its situational awareness and / or feedback from its components and / or based on information from the cloud 2104 (FIG. 17).

[0152] 22 is a logic flow diagram of a process 4000 illustrating a control program or logic configuration for correlating visualization data and instrument data in accordance with at least one aspect of the present disclosure. The process 4000 is generally performed during a surgical procedure and includes receiving or deriving 4001 a first data set, i.e., visualization data, indicative of a visual aspect of a surgical instrument relative to a surgical field, receiving or deriving 4002 a second data set, i.e., instrument data, indicative of a functional aspect of the surgical instrument, and correlating 4003 the first and second data sets.

[0153] In at least one example, correlating the visualization data with the instrument data is implemented by constructing a combined dataset from the visualization data and the instrument data. Process 4000 may further include comparing the combined dataset to another combined dataset, which may be received from an external source and / or may be derived from a previously collected combined dataset. In at least one example, process 4000 includes displaying a comparison of the two combined datasets, as described in more detail below.

[0154] The visualization data of process 4000 may represent a visual aspect of the end effector of a surgical instrument relative to tissue within a surgical field. Additionally or alternatively, the visualization data may represent a visual aspect of tissue being treated by the end effector of a surgical instrument. In at least one example, the visualization data represents one or more positions of the end effector or components thereof relative to tissue within the surgical field. Additionally or alternatively, the visualization data may represent one or more movements of the end effector or components thereof relative to tissue within the surgical field. In at least one example, the visualization data represents one or more changes in shape, size, and / or color of tissue being treated by the end effector of a surgical instrument.

[0155] In various embodiments, the visualization data is derived from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108). The visualization data can be derived from various measurements, readings, and / or any other suitable parameters monitored and / or captured by the surgical visualization system, as described in more detail in connection with FIGS. 1-18 . In various examples, the visualization data indicates one or more visual aspects of tissue within a surgical field and / or one or more visual aspects of a surgical instrument relative to tissue within the surgical field. In particular examples, the visualization data represents or identifies the position and / or movement of an end effector of a surgical instrument relative to a critical structure in the surgical field (e.g., critical structure 101 of FIG. 1 ). In particular examples, the visualization data is derived from surface mapping data, imaging data, tissue identification data, and / or distance data calculated by surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, 141.

[0156] In at least one example, the visualization data is derived from tissue identification and geometric surface mapping performed by visualization system 100 in combination with distance sensor system 104, as described in more detail in connection with Figure 1. In at least one example, the visualization data is derived from measurements, readings, or any other sensor data captured by imaging device 120. As described in connection with Figure 1, imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectrum camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to various wavelengths.

[0157] Additionally or alternatively, the visualization data may be derived from measurements, readings, or any suitable sensor data captured by any suitable imaging device, including, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In at least one example, the visualization data is derived from a visualization system 160 including an optical waveform emitter 123 and a waveform sensor 122 configured to detect a reflected waveform, as described in more detail in connection with FIGS. 3-4 and 13-16. In yet another example, the visualization data is derived from a visualization system including a three-dimensional (3D) camera and associated electronic processing circuitry, such as visualization system 500. In yet another example, the visualization data is derived from a structured (or patterned) light system 700, which is described in more detail in connection with FIG. 12. The foregoing examples may be used alone or in combination to derive the visualization data for process 4000.

[0158] The instrument data of process 4000 can be indicative of one or more operations of one or more internal components of the surgical instrument. In at least one example, the instrument data represents one or more operating parameters of the internal components of the surgical instrument. The instrument data may represent one or more positions and / or one or more movements of the one or more internal components of the surgical instrument. In at least one example, the internal component is a cutting member configured to cut tissue during a firing sequence of the surgical instrument. Additionally or alternatively, the internal component can include one or more staples configured to be fired into tissue during a firing sequence of the surgical instrument.

[0159] In at least one example, the instrument data represents one or more movements of one or more components of one or more drive assemblies of the surgical instrument, such as, for example, an articulation drive assembly, a closure drive assembly, a rotational drive assembly, and / or a firing drive assembly. In at least one example, the instrument data set represents one or more movements of one or more drive members of the surgical instrument, such as, for example, an articulation drive member, a closure drive member, a rotational drive member, and / or a firing drive member.

[0160] 23 is a schematic diagram of an exemplary surgical instrument 4600 for use with process 4000 that is similar in many respects to other surgical instruments or tools described by the present disclosure, such as, for example, surgical instrument 2112. For brevity, various aspects of process 4000 are described only in accordance with the present disclosure using handheld surgical instruments. However, this is not limiting. Such aspects of process 4000 may equally be implemented with robotic surgical instruments, such as, for example, surgical instrument 2117.

[0161] The surgical instrument 4600 includes multiple motors that can be activated to perform various functions. The multiple motors of the surgical instrument 4600 can be activated to cause a firing, closing, and / or articulation motion in the end effector. The firing, closing, and / or articulation motion can be transmitted to the end effector of the surgical instrument 4600, for example, via a shaft assembly. However, in other examples, a surgical instrument for use with the process 4000 can be configured to manually perform one or more of the firing, closing, and articulation motions. In at least one example, the surgical instrument 4600 includes an end effector that treats tissue by deploying staples into the tissue. In another example, the surgical instrument 4600 includes an end effector that treats tissue by applying therapeutic energy to the tissue.

[0162] In certain examples, the surgical instrument 4600 includes a firing motor 4602. The firing motor 4602 can be operably coupled to a firing motor drive assembly 4604, which can be configured to transfer firing motion generated by the firing motor 4602 to an end effector, particularly for moving a firing member in the form of an I-beam, which can include a cutting member, for example. In certain examples, the firing motion generated from the firing motor 4602 can deploy staples from a staple cartridge into tissue captured by the end effector and optionally advance the cutting member of the I-beam, for example, to cut the captured tissue.

[0163] In certain examples, the surgical instrument or tool may include a closure motor 4603. The closure motor 4603 can be operably coupled to a closure motor drive assembly 4605, which can be configured to transfer the closure motion generated by the closure motor 4603 to the end effector, specifically to displace a closure tube to close the anvil and compress tissue between the anvil and the staple cartridge. The closure motion can transition the end effector from an open configuration to an approximation configuration, for example, to capture tissue.

[0164] In certain examples, a surgical instrument or tool may include, for example, one or more articulation motors 4606 a, 4606 b. The articulation motors 4606 a, 4606 b can be operatively coupled to respective articulation motor drive assemblies 4608 a, 4608 b, which can be configured to transfer articulation motion generated by the articulation motors 4606 a, 4606 b to the end effector. In certain examples, the articulation motion can, for example, cause the end effector to articulate relative to the shaft.

[0165] In certain examples, a surgical instrument or tool may include a control module 4610 that can be employed with multiple motors of the surgical instrument 4600. Each of the motors 4602, 4603, 4606a, 4606b may include a torque sensor to measure the output torque on the shaft of the motor. The force on the end effector may be sensed in any conventional manner, such as by a force sensor outside the jaws or by a torque sensor on the motor that actuates the jaws.

[0166] 23, the control module 4610 may include a motor driver 4626, which may include one or more H-bridge FETs. The motor driver 4626 may modulate power delivered from a power supply 4628 to a motor coupled to the control module 4610, for example, based on input from a microcontroller 4620 ("controller"). In certain examples, as described above, the controller 4620 may be employed to determine, for example, the current drawn by a motor while the motor is coupled to the control module 4610.

[0167] In particular examples, the microcontroller 4620 may include a microprocessor 4622 (“processor”) and one or more non-transitory computer-readable media or memory units 4624 (“memory”). In particular examples, the memory 4624 can store various program instructions that, when executed, cause the processor 4622 to perform multiple functions and / or calculations described herein. In particular examples, one or more of the memory units 4624 can be coupled to the processor 4622, for example. In various examples, the processor 4622 can control a motor driver 4626 to control the position, direction of rotation, and / or speed of a motor coupled to the control module 4610.

[0168] In certain examples, one or more mechanisms and / or sensors, such as, for example, sensor 4630, can be configured to detect a force applied by the jaws of an end effector of the surgical instrument 4600 to tissue captured therebetween (the closure force "FTC"). The FTC can be transmitted to the jaws of the end effector via the closure motor drive assembly 4605. Additionally or alternatively, the sensor 4630 can be configured to detect a force applied to the end effector through the firing motor drive assembly 4604 (the firing force "FTF"). In various examples, the sensor 4630 can be configured to sense closure actuation (e.g., motor current and FTC), firing actuation (e.g., motor current and FTF), articulation (e.g., angular position of the end effector), and shaft and / or end effector rotation.

[0169] One or more aspects of process 4000 can be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4000 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4000. Additionally or alternatively, one or more aspects of process 4000 can be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, process 4000 can be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0170] In various aspects, process 4000 can be implemented by a computer-implemented interactive surgical system 2100 ( FIG. 19 ) including one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include the remote server 2113. Control circuitry performing one or more aspects of process 4000 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108) and can communicate with and receive instrument data from a surgical instrument (e.g., surgical instrument 2112, 4600). Communication between the surgical instrument and the control circuitry of the visualization system can be direct communication, or instrument data can be sent to the visualization system through the surgical hub 2106, for example. In at least one example, control circuitry performing one or more aspects of process 4000 can be a component of the surgical hub 2106.

[0171] 24 , in various examples, visualization data 4010 is correlated with instrument data 4011 by constructing a composite dataset 4012 from the visualization data 4010 and the instrument data 4011. FIG. 24 illustrates, in graph 4013, the current user's composite dataset 4012 constructed from the current user's visualization data 4010 and the current user's instrument data 4011. Graph 4013 illustrates visualization data 4010 representing a first use cycle of the surgical instrument 4600, involving jaw positioning, clamping, and firing of the surgical instrument 4600. Graph 4013 also illustrates visualization data 4010 representing the start of a second use cycle of the surgical instrument 4600, where the jaws are repositioned for a second clamping and firing of the surgical instrument 4600. Graph 4013 further illustrates the current user's instrument data 4011 in the form of FTC data 4014, which correlates with the clamping visualization data, and FTF data 4015, which correlates with the firing visualization data.

[0172] As described above, the visualization data 4010 may be derived from a visualization system (e.g., visualization system 100, 160, 500, 2108) and may represent, for example, distances between the end effector of the surgical instrument 4600 and critical structures within a surgical field during positioning, clamping, and / or firing of the end effector of the surgical instrument 4600. In at least one example, the visualization system identifies the end effector or components thereof within the surgical field, identifies critical structures within the surgical field, and tracks the position of the end effector or components thereof relative to the critical structures or tissue surrounding the critical structures. In at least one example, the visualization system identifies the jaws of the end effector within the surgical field, identifies critical structures within the surgical field, and further tracks the position of the jaws relative to the critical structures or tissue surrounding the critical structures within the surgical field.

[0173] In at least one example, the critical structure is a tumor. To remove this tumor, a surgeon will typically want to cut tissue along a safety margin around the tumor to ensure the entire tumor is removed. In such an example, the visualization data 4010 may represent the distance between the jaws of the end effector and the tumor safety margin during positioning, clamping, and / or firing of the surgical instrument 4600.

[0174] The process 4000 may further include comparing the current user's composite dataset 4012 to another composite dataset 4012′, which may be received from an external source and / or derived from a previously collected composite dataset. Graph 4013 illustrates a comparison between the current user's composite dataset 4012 and another composite dataset 4012′, which includes visualization data 4010′ and instrument data 4011′ with FTC data 4014′ and FTF data 4015′. This comparison may be presented in real time to a user of the surgical instrument 4600 in the form of graph 4013 or any other suitable format. The control circuitry executing one or more aspects of the process 4000 may compare the two composite datasets on any suitable screen in the operating room, such as, for example, a visualization system screen. In at least one example, the comparison may be displayed alongside real-time video of the surgical field captured on any suitable screen in the operating room. In at least one example, the control circuitry is configured to adjust instrument parameters to address deviations detected between the first and second composite datasets.

[0175] Additionally, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4000 may cause the current status of instrument data, such as FTF and / or FTC data, to be displayed relative to a best practice equivalent. In the example shown in FIG. 24 , the current value of FTC, represented by circle 4020, is shown in real time relative to gauge 4021, with indicator 4022 representing the best practice FTC. Similarly, the current value of FTF, represented by circle 4023, is shown relative to gauge 4024, with indicator 4025 representing the best practice FTF. Such information can be overlaid in real time on video footage of the surgical field.

[0176] 24 alerts the user that the current FTC is higher than the best practice FTC and that the current FTF is higher than the best practice FTF. The control circuitry performing one or more aspects of the process 4000 can alert the current user of the surgical instrument 4600 with an audible, visual, and / or tactile warning mechanism if the current values ​​of the FTF and / or FTC reach and / or move beyond a predetermined threshold.

[0177] In certain examples, the control circuitry (e.g., block control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of the process 4000 can further provide predicted instrument data to the current user of the surgical instrument 4600 based on the current instrument data. For example, as shown in FIG. 24, a predicted FTF 4015″ is determined based on the current value of the FTF and is further displayed on the graph 4013 for the current FTF 4015 and the previously collected FTF. Additionally or alternatively, as shown in FIG. 24, a projected FTF circle 4026 can be displayed for the gauge 4024.

[0178] In various aspects, previously collected composite data sets and / or best practice FTF and / or FTC are determined from previous uses of the surgical instrument 4600 in the same and / or other surgical procedures performed by the user, other users within the hospital, and / or users at other hospitals. Such data can be made available to the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of the process 4000, for example, by importing it from the cloud 104.

[0179] In various aspects, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4000 can visually overlay feedback measurements of tissue thickness, compression, and stiffness onto a screen displaying a live video of the surgical instrument 4600 at the surgical field as the jaws of the end effector begin to deform the tissue captured therebetween during the clamping phase. The visual overlay correlates visualized data representing tissue deformation with changes in clamping force over time. This correlation can help a user confirm proper cartridge selection, determine firing start times, and determine preferred firing rates. Additionally, the correlation can inform adaptive clamping algorithms. Adaptive firing rate changes can be signaled by changes in measured forces and tissue motion (e.g., principal strains, tissue slippage, etc.) immediately adjacent to the jaws of the surgical instrument 4600, while gauges or meters communicating the results are superimposed on a screen displaying a live video of the end effector at the surgical field.

[0180] In addition to the above, the kinematics of the surgical instrument 4600 can also be used to direct the motion of the instrument to another use or user. The kinematics can be ascertained with an accelerometer, torque sensor, force sensor, motor encoder, or any other suitable sensor to obtain various force, velocity, and / or acceleration data of the surgical instrument or its components and correlate with corresponding visualization data.

[0181] In various aspects, if the visualization data and / or instrument data detect a deviation from best practice surgical technique, the control circuitry executing one or more aspects of process 4000 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can suggest an alternative surgical technique. In at least one example, an adaptive display of instrument movement, force, tissue impedance, and results from the proposed alternative technique is shown. If the visualization data indicates detection of a vessel and clip applier within the surgical field, the control circuitry executing one or more aspects of process 4000 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can further ensure perpendicularity of the vessel relative to the clip applier. The control circuitry can suggest changes in position, orientation, and / or roll angle to achieve the desired perpendicularity.

[0182] In various aspects, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can perform process 4000 by comparing real-time visualization data with a pre-operative planning simulation. A user can utilize a pre-operative patient scan to simulate a surgical approach. The pre-operative planning simulation can allow a user to follow a particular pre-operative plan based on a training run. The control circuitry can be configured to correlate fiducial landmarks from the pre-operative scan / simulation with the current visualization data. In at least one example, the control circuitry may employ object boundary tracking to establish the correlation.

[0183] As a surgical instrument interacts with tissue and deforms its surface geometry, the change in surface geometry can be calculated as a function of the surgical instrument's position. For a given change in surgical instrument position upon contact with tissue, the corresponding change in tissue geometry may depend on the subsurface structure within the tissue region contacted by the surgical instrument. For example, in thoracic surgery, the change in tissue geometry in regions containing airway substructures will be different from regions with parenchymal tissue substructures. Generally, stiffer substructures will result in a smaller change in surface tissue geometry in response to a given change in surgical instrument position. In various aspects, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to calculate a running average of the change in surgical instrument position relative to changes in surface geometry for a given patient to obtain patient-specific variances. Additionally or alternatively, the calculated running average can be compared to a second set of previously collected data. In certain examples, a surface reference may be selected when a change in surface geometry is not measured for each change in instrument position. In at least one example, the control circuitry may be configured to determine the position of the substructure based on the detected change in surface geometry in response to a given contact between the tissue region and the surgical instrument.

[0184] Additionally, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to maintain pre-set instrument-tissue contact throughout tissue treatment based on a correlation between the pre-set instrument-tissue contact and one or more tissue and surface geometry changes associated with the pre-set instrument-tissue contact. For example, the end effector of the surgical instrument 4600 can clamp tissue between its jaws at a desired compression that establishes instrument-tissue contact. Corresponding changes in tissue and surface geometry can be detected by a visualization system. Additionally, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can derive visualization data indicative of the changes in tissue and surface geometry associated with the desired compression. Additionally, the control circuitry can automatically adjust motor settings of the closure motor 4603 (FIG. 22) to maintain the changes in tissue and surface geometry associated with the desired compression. This arrangement requires continuous interaction between the surgical instrument 4600 and the visualization system to maintain the changes in tissue and surface geometry associated with the desired compression by continuously adjusting the compression of the jaws against the tissue based on visualization data.

[0185] In yet another example, if the surgical instrument 4600 is a robotic tool attached to a robotic arm of a robotic surgical system (e.g., robotic system 110), the robotic surgical system can be configured to automatically adjust one or more components of the robotic surgical system to maintain the set surface contact with the tissue based on visualization data derived from detected changes in the surface geometry of the tissue detected in response to the set surface contact with the tissue.

[0186] In various examples, visualization data can be used in combination with measured instrument data to maintain tissue-to-tissue contact, either through position or load control, allowing the user to manipulate the tissue and apply a predetermined load to the tissue while moving the instrument relative to the tissue. The user can specify whether they want to maintain contact or pressure, and visual tracking of the instrument along with the internal load of the instrument can be used to allow repositioning without changing fixed parameters.

[0187] 25A and 25B , a screen 4601 of a visualization system (e.g., visualization system 100, 160, 500, 2108) displays real-time video footage of a surgical field during a surgical procedure. The end effector 4642 of the surgical instrument 4600 includes jaws that clamp tissue near a tumor identified in the surgical field, for example, via a registered MRI image. The jaws of the end effector 4642 include an anvil 4643 and a channel that houses a staple cartridge. At least one of the anvil 4643 and the channel is movable relative to the other to capture tissue between the anvil 4643 and the staple cartridge. The captured tissue is then stapled via staples 4644 deployable from the staple cartridge during the firing sequence of the surgical instrument 4600. Additionally, the captured tissue is cut via a distally advanced cutting member 4645 during the firing sequence, but shortly after staple deployment.

[0188] 25A , during the firing sequence, the position and / or movement of captured tissue and certain internal components of the end effector 4642, such as the staples 4644 and cutting member 4645, may not be visible in the standard view 4640 of the live feed on the screen 4601. Certain end effectors include windows 4641, 4653 that allow the cutting member 4645 to be partially visible at the beginning and end of the firing sequence, but not during the firing sequence. Thus, a user of the surgical instrument 4600 is unable to track the progress of the firing sequence on the screen 4601.

[0189] 26 is a logic flow diagram of a process 4030 illustrating a control program or logic configuration for synchronizing movement of a virtual representation of an end effector component with actual movement of the end effector component, according to at least one aspect of the present disclosure. The process 4030 is generally executed during a surgical procedure and includes detecting movement of an internal component of the end effector during a firing sequence at 4031, presenting a virtual representation of the internal component on the end effector at 4032, e.g., by superimposing it, and synchronizing movement of the virtual representation on a screen 4601 with the detected movement of the internal component at 4033.

[0190] One or more aspects of process 4030 may be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4030 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4030. Additionally or alternatively, one or more aspects of process 4030 may be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, process 4030 may be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0191] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4030 can receive instrument data indicative of the movement of internal components of the end effector 4642 during a firing sequence. The movement of the internal components can be tracked using, for example, a conventional rotary encoder on the firing motor 4602. In other examples, the movement of the internal components can be tracked by a tracking system employing an absolute positioning system. A detailed description of absolute positioning systems is provided in U.S. Patent Application Publication No. 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In some examples, the movement of the internal components can be tracked using one or more position sensors, which can include any number of magnetic sensing elements, such as, for example, magnetic sensors classified according to whether they measure the total magnetic field or vector components of the magnetic field.

[0192] In various aspects, the process 4030 includes an overlay trigger. In at least one example, the overlay trigger can detect when tissue has been captured by the end effector 4642. When tissue captured by the end effector 4642 is detected, the process 4030 superimposes a virtual representation of the cutting member 4645 in a start position onto the end effector 4642. The process 4030 further includes projecting a staple line outlining where staples will be deployed within the captured tissue. Additionally, in response to a user actuating a firing sequence, the process 4030 moves the virtual representation of the cutting member 4645 distally, mimicking the actual movement of the cutting member 4645 within the end effector 4642. Once the staples are deployed, the process 4030 converts the unfired staples into fired staples, allowing the user to visually track the staple deployment and advancement of the cutting member 4645 in real time.

[0193] In various examples, the control circuitry (e.g., block control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4030 can detect that tissue has been captured by the end effector 4642, for example, from instrument data indicating the force that the closure motor 4603 (FIG. 22) applies to the jaws of the end effector 4642 through the closure motor drive assembly 4605. The control circuitry can further determine the position of the end effector within the surgical field from visualization data derived from a visualization system (e.g., visualization system 100, 160, 500, 2108). In at least one example, the position of the end effector can be determined relative to a reference point within the tissue, such as, for example, a critical structure.

[0194] In either case, the control circuitry superimposes a virtual representation of the internal components onto the end effector 4642 on the screen 4601 at a location corresponding to the location of the internal components within the end effector. Additionally, the control circuitry moves the projected virtual representation of the internal components in synchronization with the internal components during the firing sequence. In at least one example, synchronization is improved by the integration of markings on the end effector 4642 that the control circuitry can use as reference points to determine where to superimpose the virtual representations.

[0195] FIG. 25B shows an augmented view 4651 of a live feed of the surgical field on the screen 4601. In the example augmented view 4651 of FIG. 25B, virtual representations of staples 4644 and cutting element 4645 are superimposed on the end effector 4642 during the firing sequence. The overlay tracks the progress of the firing sequence, distinguishing between fired staples 4644a and unfired staples 4644b and between completed cut lines 4646a and projected cut lines 4646b. The overlay also shows the start of staple line 4647 and the projected end 4649 of the staple line for cut lines that do not reach the tissue edge. Based on the overlay of the tumor MRI image, a safety margin distance "d" between the tumor and projected cut line 4646b is measured and presented along with the overlay. The superimposed safety margin distance "d" assures the user that all of the tumor will be removed.

[0196] As shown in Figure 25B, the control circuitry is configured to cause the visualization system to continuously reposition the virtual representation of the internal components to correlate with the actual movement of the internal components. In the example of Figure 25B, this overlay shows the completed cut line 4646 lagging slightly behind the fired staple line 4644a by a distance "d1," assuring the user that the firing sequence is proceeding properly.

[0197] 27, a visualization system (e.g., visualization system 100, 160, 500, 2108) can detect and / or define trocar location employing instrument illumination 4058 and camera 4059. Based on the determined trocar location, the user can be directed to the most suitable trocar port to complete the intended function based on time efficiency, location of critical structures, and / or avoidance or risk.

[0198] FIG. 27 shows three trocar positions (trocar 1, trocar 2, and trocar 3) extending through a body wall 4050 at different positions and orientations relative to the body wall and relative to a critical structure 4051 within a cavity 4052 within the body wall 4050. The trocars are represented by arrows 4054, 4055, and 4056. A lighting fixture 4058 can detect the trocar position using cascading light or images on the surrounding environment. Additionally, the light source of the lighting fixture 4058 can be a rotating light source. In at least one example, the light sources of the lighting fixture 4058 and camera 4059 are utilized to detect the distance of the trocar relative to a target location, such as the critical structure 4051. In various examples, the visualization system can suggest a change in the position of the instrument once a more favorable instrument position is determined based on visualization data derived from a camera 4059 recording the light projected by the light source of the lighting fixture 4058. The screen 4060 can display the distance between the trocar and the target tissue, whether instrument access through the trocar is acceptable, the risks associated with using the trocar, and / or the expected surgical time using the trocar, which can assist the user in selecting the optimal trocar for introducing surgical tools into the cavity 4052.

[0199] In various aspects, a surgical hub (e.g., surgical hub 2106, 2122) can recommend an optimal trocar for inserting a surgical tool into cavity 4052 based on user characteristics, which may be received, for example, from a user database. User characteristics include the user's handedness, patient-user preferences, and / or the user's physical characteristics (e.g., height, arm length, range of motion). The surgical hub can utilize these characteristics, available trocar location and orientation data, and / or critical structure location data to select the optimal trocar for inserting the surgical tool to reduce user fatigue and increase efficiency. In various aspects, the surgical hub can also reverse its control over the surgical instrument when the user reverses the orientation of the end effector.

[0200] The surgical hub can reconfigure the output of the surgical instrument based on the visualization data, for example, the surgical hub can inhibit activation of the therapeutic energy output of the surgical instrument if the visualization data indicates that the surgical instrument is being retracted or utilized to perform a different task.

[0201] In various aspects, the visualization system can be configured to track the blood surface or estimate blood volume-based reflected IR or red wavelengths to delineate blood from non-blood surface and surface geometry measurements, which can be reported as absolute static measures or as rates of change to provide quantitative data on the amount and degree of change of bleeding.

[0202] Referring to FIG. 28, various elements of a visualization system (e.g., visualization system 100, 160, 500, 2108), such as a structured light projector 706 and a camera 720, can be used to generate visualization data of an anatomical organ to generate a virtual 3D structure 4130 of the anatomical organ.

[0203] As described herein, structured light, for example in the form of stripes or lines, can be projected from a light source and / or projector 706 onto a surface 705 of a targeted anatomical structure to identify the shape and contours of the surface 705. For example, a camera 720, which may be similar in many ways to imaging device 120 (FIG. 1), can be configured to detect the projected light pattern on the surface 705. The deformation of the projected pattern upon impacting the surface 705 allows the vision system to calculate depth and surface information of the targeted anatomical structure.

[0204] 29 is a logic flow diagram of a process 4100 illustrating a control program or logic configuration according to at least one aspect of the present disclosure. In various examples, the process 4100 identifies a surgical procedure at 4101 and identifies an anatomical organ targeted by the surgical procedure at 4102. The process 4100 further generates a virtual 3D structure 4130 of at least a portion of the anatomical organ at 4104, identifies anatomical structures of at least a portion of the anatomical organ associated with the surgical procedure at 4105, connects the anatomical structures to the virtual 3D structure 4130 at 4106, and overlays a layout plan of the surgical procedure, determined based on the anatomical structures, onto the virtual 3D structure 4130 at 4107.

[0205] One or more aspects of process 4100 can be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4100 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4100. Additionally or alternatively, one or more aspects of process 4100 can be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, one or more aspects of process 4100 can be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0206] In various aspects, process 4100 can be implemented by a computer-implemented interactive surgical system 2100 ( FIG. 19 ) that includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 that may include the remote server 2113. Control circuitry that performs one or more aspects of process 4100 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108).

[0207] Control circuitry (e.g., block control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4100 can identify 4101 the surgical procedure and / or identify 4102 the anatomical organ targeted by the surgical procedure by obtaining such information from a database storing such information or by obtaining the information directly from user input. In at least one example, the database is stored in a cloud-based system (e.g., cloud 2104, which may include a remote server 2113 coupled to storage device 2105). In at least one example, the database includes an in-hospital EMR.

[0208] In one aspect, the surgical system 2200 includes a surgical hub 2236 connected to multiple surgical field devices, such as, for example, a visualization system (e.g., visualization system 100, 160, 500, 2108) located in an operating room. In at least one example, the surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the visualization system, the cloud 2204, and / or a remote server 2213. The control circuitry of the surgical hub 2236 performing one or more aspects of the process 4100 can identify 4101 the surgical procedure and / or identify 4102 the anatomical organ targeted by the surgical procedure by retrieving such information from a database stored on the cloud 2204 or from the remote server 2213.

[0209] Control circuitry executing one or more aspects of process 4100 may cause a visualization system (e.g., visualization system 100, 160, 500, 2108) to perform an initial scan of at least a portion of an anatomical organ to generate 4104 a three-dimensional ("3D") structure 4130 of at least a portion of the anatomical organ targeted for surgery. In the example shown in FIG. 28 , the anatomical organ is a stomach 4110. The control circuitry may cause one or more elements of the visualization system, such as structured light projector 706 and camera 720, to generate visualization data by performing a scan of at least a portion of the anatomical organ using structured light 4111 when the camera(s) are introduced into the body. The 3D structure of at least a portion of the anatomical organ can be generated utilizing current visualization data, pre-operative data (e.g., patient scans and other relevant clinical data), visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input.

[0210] Further, the control circuitry executing one or more aspects of process 4100 identifies, at 4105, an anatomical structure of at least a portion of the anatomical organ relevant to the surgical procedure. In at least one example, a user can select the anatomical structure using any suitable input device. Additionally or alternatively, the visualization system can include one or more imaging devices 120 with a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectrum camera) configured to detect reflected spectral waveforms and generate an image based on molecular responses to various wavelengths. Optical absorption or refractive properties of tissue can be utilized by the control circuitry to distinguish between different tissue types of the anatomical organ and thereby identify the relevant anatomical structure. Additionally, the control circuitry can utilize current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), stored visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input to identify the relevant anatomical organ.

[0211] The identified anatomical structures may be anatomical structures of the surgical field, and / or the anatomical structures may be selected by a user. In various examples, positional tracking of relevant anatomical structures may be extended beyond the current view of a camera directed at the surgical field. In one example, this is achieved by using common visible connected landmarks or by using secondary connected motion tracking. Secondary tracking may be achieved, for example, through a secondary imaging source, calculation of scope motion, and / or pre-established beacons measured by a secondary visualization system.

[0212] As described in detail above in connection with FIG. 14 , the visualization system can utilize a structured light projector 706 to cast an array of patterns or lines from which the camera 720 can determine the distance to the target location. The visualization system can then project a pattern or line of known size at a set distance equal to the determined distance. In addition, a spectral camera can determine the size of the pattern, which can vary depending on the optical absorption or refractive properties of tissue at the target location. The difference between the known size and the determined size indicates the tissue density at the target location, which in turn indicates the tissue type at the target location. Control circuitry performing one or more aspects of process 4100 can identify relevant anatomical structures based at least in part on the determined tissue density at the target location.

[0213] In at least one example, detected abnormalities in tissue density may be associated with a disease state. Furthermore, the control circuitry selects, updates, or modifies one or more settings of a surgical instrument treating the tissue based on the tissue density detected via the visualization data. For example, the control circuitry can alter various clamping and / or firing parameters of a surgical stapler utilized to staple and cut the tissue. In at least one example, the control circuitry can slow down the firing sequence and / or enable an extended clamp time based on the tissue density detected via the visualization data. In various examples, the control circuitry can alert a user of the surgical instrument to abnormal tissue density by displaying on a screen, for example, instructions to reduce bite size, increase or decrease the energy delivery output of the electrosurgical instrument, and adjust the amount of jaw closure. In another example, if the visualization data indicates the tissue is fatty tissue, the instruction may be to increase power to shorten the energy application time.

[0214] Furthermore, identifying the type of surgical procedure can facilitate target organ identification by the control circuitry. For example, if the procedure is a left upper lobectomy, the lung is likely the target organ. Thus, the control circuitry considers only visible and non-visual data related to the lung and / or instruments commonly used in such procedures. Knowledge of the procedure type then makes other image fusion algorithms more effective, for example, to inform tumor location or staple line placement.

[0215] In various aspects, knowledge of the operating table position and / or pneumoperitoneum pressure can be used by control circuitry executing one or more aspects of process 4100 to establish a baseline position of the target anatomical organ and / or associated anatomical structures identified from the visualization data. Movement of the operating table (e.g., moving the patient from a flat position to a reverse Trendelenburg position) can cause deformation of the anatomical structures, which can be tracked and compared to the baseline to continually inform the position and status of the target organ and / or associated anatomical structures. Similarly, changes in pneumoperitoneum pressure within the body cavity can interfere with the baseline visualization data of the target organ and / or associated anatomical structures within the body cavity.

[0216] The control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more aspects of a process to derive baseline visualization data of a patient's target organ and / or associated anatomical structures on an operating table during a surgical procedure, determine changes in the position of the operating table, and re-derive baseline visualization data of the patient's target organ and / or associated anatomical structures in the new position.

[0217] Similarly, the control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more aspects of a process to derive baseline visualization data of a patient's target organs and / or associated anatomical structures on an operating table during a surgical procedure, determine changes in pneumoperitoneum pressure within the patient's body cavity, and re-derive baseline visualization data of the patient's target organs and / or associated anatomical structures at the new pneumoperitoneum pressure.

[0218] In various examples, as shown in FIG. 28 , the control circuitry performing one or more aspects of process 4100 can link the identified anatomical structures to the virtual 3D structure by overlaying landmarks or markers on the virtual 3D structure of the organ to indicate the location of the anatomical structures. The control circuitry can also overlay user-defined structure and tissue planes on the virtual 3D structure. In various aspects, a hierarchy of tissue types can be established to organize the identified anatomical structures on the virtual 3D structure. Table 1, provided below, shows an example hierarchy for the lung and stomach.

[0219] [Table 1]

[0220] In various aspects, the relevant anatomical structures identified on the virtual 3D structure can be renamed and / or relocated by the user to correct errors or as desired. In at least one example, the corrections can be voice-activated. In at least one example, the corrections are recorded for future machine learning purposes.

[0221] Further to the above, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4100 can superimpose a surgical layout plan (e.g., layout plan 4120) onto a virtual 3D structure of the target organ (e.g., stomach 4110) at 4107. In at least one example, the virtual 3D structure is displayed on a separate screen of the visualization system from the screen displaying a live feed / view of the surgical field. In another example, one screen can alternate between displaying the surgical field and a live feed of the 3D structure. In such an example, a user can alternate between the two views using any suitable input device.

[0222] 28, the control circuitry determines that the surgical procedure is a sleeve gastrectomy and the target organ is the stomach. In an initial scan of the abdominal cavity, the control circuitry uses visualization data, such as structured light data and / or spectral data, to identify the stomach, liver, spleen, greater curvature of the stomach, and pylorus, informed by knowledge of the procedure and target structures.

[0223] For example, visualization data such as structured light data and / or spectral data can be utilized by the control circuitry to identify the stomach 4110 by comparing current structured light data with stored structured light data previously associated with such organ. In at least one embodiment, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can utilize structured light data representative of characteristic anatomical contours of organs and / or spectral data representative of characteristic subsurface tissue features to identify anatomical structures relevant to a surgical layout plan (e.g., layout plan 4120).

[0224] In at least one example, the visualization data can be used to identify the pyloric vein 4112, which indicates the location 4113 of the pylorus 4131; the gastro-colic vessels 4114, which indicates the location 4115 of the greater curvature 4110 of the stomach; the curvature 4116 of the right gastric vein, which indicates the location 4117 of the incisor angle 4132; and / or the location 4119 of the angle of His 4121. The control circuitry can assign landmarks to one or more of the identified locations. In at least one example, as shown in FIG. 28 , the control circuitry causes the visualization system to overlay landmarks on the locations 4113, 4117, 4119 on a virtual 3D structure of the stomach 4110 generated using the visualization data as described above. In various aspects, the landmarks can be synchronously overlaid on the virtual 3D structure and the surgical field view, allowing the user to switch views without losing track of the landmarks. The user can zoom in on the view of the screen displaying the virtual 3D structure to show the overall layout plan or can zoom in to show a portion similar to the surgical field view. The control circuitry can continuously track and update the landmarks.

[0225] In a sleeve gastrectomy, the surgeon primarily staples stomach tissue approximately 4 cm from the pylorus. Prior to stapling, an energy device is introduced into the patient's abdominal cavity at the start of the sleeve gastrectomy procedure, and the gastroepiploic artery and peritoneum are incised from the greater curvature at or about 4 cm from the pylorus. As described above, the control circuitry that has identified position 4113 can automatically cause the overlay of the end effector of the energy device at or about 4 cm from position 4113. The start position of the sleeve gastrectomy can be identified by overlaying the end effector of the energy device at or about 4 cm from the pylorus, or any suitable landmark.

[0226] As the surgeon incises along the greater curvature of the stomach, the control circuitry causes the removal of landmarks at location 4113 and / or the superimposed end effector of the energy device. As the surgeon approaches the spleen, a distance indicator is automatically superimposed on the virtual 3D structure view and / or surgical field view. The control circuitry can cause the distance indicator to identify a distance of 2 cm from the spleen. The control circuitry can cause the distance indicator to flash and / or change color when the incision path reaches or is about to reach, for example, 2 cm from the spleen. The distance indicator overlay remains until the user reaches location 4119 of the His angle 4121.

[0227] Referring to FIG. 30 , as the surgical stapler is introduced into the abdominal cavity, the control circuitry can utilize the visualization data to identify the pylorus 4131, the angular notch 4132, the greater curvature 4133 of the stomach 4110, the lesser curvature 4134 of the stomach 4110, and / or other anatomical structures relevant to the sleeve gastrectomy procedure. A bougie overlay may be similarly shown. The introduction of a surgical instrument into the body cavity (e.g., the introduction of the surgical stapler into the abdominal cavity) can be detected by the control circuitry from the visualization data indicating visual cues on the end effector, such as a unique color, marking, and / or shape. The control circuitry can identify the surgical instrument in a database storing such visual cues and corresponding visual cues. Alternatively, the control circuitry can prompt the user to identify the surgical instrument inserted into the body cavity. Alternatively, a surgical trocar that aids in access to the body cavity can include one or more sensors for detecting a surgical instrument inserted therethrough. In at least one example, the sensor comprises an RFID reader configured to identify the surgical instrument from an RFID chip on the surgical instrument.

[0228] In addition to landmarks identifying relevant anatomical structures, the visualization system can also superimpose a surgical procedure layout plan 4135, which may be in the form of a recommended treatment path, onto the 3D critical structures and / or onto the surgical field view. In the example of Figure 30, the surgical procedure is a sleeve gastrectomy, and the procedure layout plan 4135 is in the form of three resection paths 4136, 4137, 4138 and the corresponding outcome volumes of the resulting sleeves.

[0229] As shown in FIG. 30 , the starting points are distances (a, a1, a2) from the pylorus 4131 to create the sleeve. Each starting point results in a different sleeve size (e.g., 400 cc, 425 cc, and 450 cc for starting points 4146, 4147, and 4148 at distances a, a1, and a2 from the pylorus 4131, respectively). In one example, the control circuitry prompts the user for a size selection and, in response, provides the selected sleeve size and presents a treatment layout plan, which may be in the form of an ablation path. In another example, as shown in FIG. 30 , the control circuitry presents multiple ablation paths 4136, 4137, and 4138 and corresponding sleeve sizes. The user can then select one of the proposed ablation paths 4136, 4137, and 4138, and in response, the control circuitry removes the non-selected ablation path.

[0230] In yet another example, the control circuitry allows the user to adjust the proposed resection path on a screen showing the virtual 3D structure and / or the resection path superimposed on the surgical field. The control circuitry can calculate the sleeve size based on the adjustment. Or, in another example, the user can select a starting point for forming the sleeve at a desired distance from the pylorus 4131. In response, the control circuitry calculates the sleeve size based on the selected starting point.

[0231] For example, presentation of a resection path can be achieved by having the visualization system overlay the resection path on the virtual 3D structure view and / or the surgical field view, and conversely, removal of a proposed resection path can be achieved by having the visualization system remove such resection path overlay from the virtual 3D structure view and / or the surgical field view.

[0232] 30 , in certain embodiments, when the end effector of the surgical stapler clamps the stomach tissue between a start point selected from the proposed start points 4146, 4147, 4147 and an end position 4140 that is a predetermined distance from the incisional angle 4132, the control circuitry presents information related to the clamping and / or firing of the surgical stapler. In at least one example, as shown in FIG. 24 , a combined data set 4012 can be displayed from the visualization data 4010 and the instrument data 4011. Additionally or alternatively, values ​​of the FTC and / or FTF can be displayed. For example, a current value of the FTC, represented by circle 4020, can be shown in real time relative to gauge 4021, and indicator 4022 can represent the best performed FTC. Similarly, a current value of the FTF, represented by circle 4023, can be shown relative to gauge 4024, and indicator 4025 can represent the best performed FTF.

[0233] After the surgical stapler is fired, recommendations for selecting a new cartridge can be presented on the surgical stapler's screen or on one of the visualization system's screens, as described in more detail below. When the surgical stapler is removed from the abdominal cavity, reloaded with the selected staple cartridge, and reintroduced into the abdominal cavity, distance indicators identifying a fixed distance (d) from multiple points along the lesser curvature 4134 of the stomach 4110 to the selected resection path are superimposed on the virtual 3D structure view and / or the surgical field view. To ensure proper orientation of the surgical stapler's end effector, the distance of the distal end of the surgical stapler's end effector from the target and the distance from the proximal end to the previously fired staple line are superimposed on the virtual 3D structure view and / or the surgical field view. This process is repeated until the resection is complete.

[0234] One or more of the distances suggested and / or calculated by the control circuitry can be determined based on stored data, which in at least one example includes pre-operative data, user preference data, and / or data from surgical procedures previously performed by the user or other users.

[0235] 31 , process 4150 illustrates a control program or logic configuration for proposing a resection path for removing a portion of an anatomical organ according to at least one aspect of the present disclosure. Process 4150 identifies an anatomical organ targeted for a surgical procedure at 4151, identifies an anatomical structure of the anatomical organ relevant to the surgical procedure at 4152, and proposes a resection path for removing a portion of the anatomical organ with a surgical instrument at 4153, as described in more detail elsewhere herein in connection with process 4100. The surgical resection path is determined based on the anatomical structure. In at least one example, the surgical resection path includes distinct starting points.

[0236] One or more aspects of process 4150 can be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4150 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4150. Additionally or alternatively, one or more aspects of process 4150 can be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, process 4150 can be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0237] 32A-32D, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4100 or process 4150 may utilize dynamic visualization data to update or modify a surgical layout plan in real time during implementation. In at least one example, the control circuitry modifies a pre-defined resection path ( FIG. 32B ) for removing a portion of an organ or abnormality (e.g., a tumor or site) with a surgical instrument to an alternative resection path ( FIG. 32D ) based on dynamic visualization data from one or more imaging devices of a visualization system (e.g., visualization system 100, 160, 500, 2108) that tracks the progress of the tissue being resected and surrounding tissue. The modification of the resection path can be triggered by a deviation in the position of a critical structure (e.g., a blood vessel) relative to the resection path. For example, the tissue ablation process can sometimes lead to inflammation of the tissue, which can change the shape and / or volume of the tissue, thereby causing critical structures (e.g., blood vessels) to shift position. The dynamic visualization data enables the control circuitry to detect position and / or volume changes of critical structures and / or associated anatomical structures near the pre-programmed ablation path. If the position and / or volume changes cause the critical structures to shift into or within a safe margin from the ablation path, the control circuitry modifies the pre-programmed ablation path by selecting or at least recommending an alternative ablation path for the surgical instrument.

[0238] FIG. 32A shows a live view 4201 of a surgical field on a visualization system screen 4230. A surgical instrument 4200 is introduced into the surgical field to remove a target region 4203. An initial planning layout 4209 for removing the region is superimposed on the live view 4201, as shown in the enlarged view of region 4203 in FIG. 32B. Region 4203 is surrounded by critical structures 4205, 4206, 4207, and 4208. As shown in FIG. 32B, the initial planning layout 4209 extends a resection path from region 4203 around region 4203 at a predetermined safety margin. The resection path avoids crossing or passing through critical structures by extending either outside (e.g., critical structure 4208) or inside (e.g., critical structure 4206) the critical structures. As described above, the initial planning layout 4209 is determined by control circuitry based on visualization data from the visualization system.

[0239] FIG. 32A shows a live view 4201′ of the surgical field on the visualization system screen 4230 at a later time (00:43). The end effector 4202 of the surgical instrument 4200 resects tissue along a predetermined resection path defined by the layout plan 4209. Volumetric changes in the tissue, including region 4203 due to tissue inflammation, cause critical structures 4206 and 4208, for example, to shift toward the predetermined resection path. In response, as shown in FIG. 32D , the control circuitry suggests an alternative resection path 4210 that navigates around the critical structures 4206, 4208 to protect them from damage. In various examples, the alternative resection path can be suggested to minimize bleeding, shorten surgical time, and balance the impact on the volume of remaining organs by providing guidance to the user to optimize the amount of healthy tissue to be spared and ensure critical structures are not hit, while also minimizing bleeding and reducing the surgical time and pressure to deal with unexpected situations.

[0240] In various aspects, control circuitry performing one or more aspects of one or more processes described by the present disclosure can receive and / or derive visualization data from multiple imaging devices of a visualization system. The visualization data can facilitate tracking of critical structures outside of the live view of the surgical field. Common landmarks can enable the control circuitry to integrate visualization data from the multiple imaging devices of the visualization system. In at least one example, for example, secondary tracking of critical structures outside of the live view of the surgical field can be achieved through a secondary imaging source, calculation of scope motion, or pre-established beacons / landmarks measured by a second system.

[0241] 33-35 , a logic flow diagram of a process 4300 illustrates a control program or logic configuration for presenting or overlaying surgical instrument parameters on or near a proposed surgical resection path according to at least one aspect of the present disclosure. Process 4300 is generally performed during a surgical procedure and includes identifying an anatomical organ targeted for the surgical procedure at 4301, identifying anatomical structures relevant to the surgical procedure from visualization data from at least one imaging device at 4302, and proposing a surgical resection path for removing a portion of the anatomical organ with a surgical instrument at 4303. In at least one example, the surgical resection path is determined based on the anatomical structures. Process 4300 further includes presenting surgical instrument parameters in accordance with the surgical resection path at 4304. Additionally or alternatively, process 4300 further includes adjusting surgical instrument parameters in accordance with the surgical resection path at 4305.

[0242] One or more aspects of process 4300 may be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4300 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4030. Additionally or alternatively, one or more aspects of process 4300 may be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, process 4300 may be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0243] In various examples, control circuitry (e.g., block control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4300 may identify an anatomical organ targeted for surgery at 4301, identify anatomical structures relevant to the surgery from visualization data from at least one imaging device of a visualization system (e.g., visualization system 100, 160, 500, 2108) at 4302, and / or propose a surgical resection path for removing a portion of the anatomical organ with a surgical instrument (e.g., surgical instrument 4600) at 4303, as described elsewhere herein in connection with processes 4150 (FIG. 31), 4100 (FIG. 29). Additionally, control circuitry performing one or more aspects of process 4300 may suggest or recommend one or more parameters of the surgical instrument according to the proposed surgical resection path at 4303. In at least one example, the control circuit presents recommended parameters for the surgical instrument by overlaying such parameters on or near the proposed surgical path at 4304, as shown in Figures 34 and 35.

[0244] FIG. 34 illustrates a virtual 3D structure 4130 of a patient's stomach undergoing a sleeve gastrectomy performed using a surgical instrument 4600, in accordance with at least one aspect of the present disclosure. As described in more detail in connection with FIG. 28 , various elements of a visualization system (e.g., visualization system 100, 160, 500, 2108), such as, for example, a structured light projector 706 and a camera 720, can be used to create visual data to generate the virtual 3D structure 4130. Relevant anatomical structures (e.g., pylorus 4131, angular notch 4132, angle of His 4121) are identified from visualization data from one or more imaging devices of the visualization system. In at least one example, landmarks are assigned to the locations 4113, 4117, 4119 of such anatomical structures by overlaying the landmarks on the virtual 3D structure 4130.

[0245] Additionally, at 4303, a surgical resection path 4312 is proposed based on the identified anatomical structures. In at least one example, as shown in FIG. 34 , the control circuitry overlays the surgical resection path 4312 on a virtual 3D structure 4130. As described in more detail elsewhere herein, the proposed surgical path can be automatically adjusted based on the desired volumetric output. Additionally, the projected margins can be automatically adjusted based on critical structures and / or tissue abnormalities automatically identified by the control circuitry from the visualization data.

[0246] In various aspects, the control circuitry executing at least one aspect of the process 4300 presents parameters 4314 of the surgical instrument selected according to the surgical resection path 4312 proposed in 4303. In the example shown in FIG. 34 , the parameters 4314 indicate a staple cartridge automatically selected for use with the surgical instrument 4600 in performing a sleeve gastrectomy based on the surgical resection path proposed in 4303. In at least one example, the parameters 4314 include at least one of a staple cartridge size, a staple cartridge color, a staple cartridge type, and a staple cartridge length. In at least one example, the control circuitry presents recommended parameters 4314 for the surgical instrument 4600 at 4304 by overlaying such parameters on or near the surgical path 4312 proposed in 4303, as shown in FIGS. 34 and 35 .

[0247] In various embodiments, the control circuitry performing at least one aspect of the process 4300 presents a tissue parameter 4315 along one or more portions of the surgical resection path 4312. In the example shown in FIG. 34 , the tissue parameter 4315 is tissue thickness presented by displaying a cross-section along line AA representing the tissue thickness along at least a portion of the surgical resection path 4312. In various embodiments, the staple cartridge utilized by the surgical instrument 4600 may be selected according to the tissue parameter 4315. For example, as shown in FIG. 34 , a black cartridge with larger staple size may be selected for use with the thicker muscle tissue of the vestibule, while a green cartridge with smaller staple size may be selected for use with the myocardial tissue of the body and the fundus of the stomach.

[0248] The tissue parameters 4315 include at least one of tissue thickness, tissue type, and sleeve volume outcome resulting from the proposed surgical resection path 4312. The tissue parameters 4315 may be derived from previously captured CT, ultrasound, and / or MRI images of the patient's organ and / or from previously known average tissue thickness. In at least one example, the surgical instrument 4600 is an intelligent instrument (similar to the intelligent instrument 2112), and the tissue thickness and / or selected staple cartridge information is transmitted to the surgical instrument 4600 to optimize closure settings, firing settings, and / or any other suitable surgical instrument settings. In one example, as described in connection with FIGS. 17-19 , the tissue thickness and / or selected staple cartridge information can be transmitted to the surgical instrument 4600 from a visualization system (e.g., visualization system 100, 160, 500, 2108) and a surgical hub (e.g., surgical hub 2106, 2122) in communication with the surgical instrument 4600.

[0249] In various examples, the control circuitry performing at least one aspect of the process 4300 suggests placements 4317 of two or more staple cartridge sizes (e.g., 45 mm and 60 mm) according to the determined tissue thickness along at least a portion of the surgical resection path 4312. Further, as shown in FIG. 35 , the control circuitry can suggest placements 4317 along the surgical resection path 4312 suggested in 4303. Alternatively, the control circuitry can suggest appropriate placements 4317 along a user-selected surgical resection path. The control circuitry can determine the tissue thickness along the user-selected resection path, as described above, and suggest staple cartridge placements according to the tissue thickness.

[0250] In various embodiments, the control circuitry executing one or more aspects of the process 4300 can suggest a surgical resection path or optimize a selected surgical resection path to minimize the number of staple cartridges in the staple cartridge arrangement 4317 without compromising the size of the resulting sleeve below a predetermined threshold. Reducing the number of used cartridges reduces procedure time and costs and reduces trauma to the patient.

[0251] 35, the arrangement 4317 includes a first staple cartridge 4352 and a final staple cartridge 4353 that define the beginning and end of the surgical resection path 4312. If only a small portion of the final staple cartridge 4353 of the proposed staple cartridge arrangement 4317 is needed, the control circuitry can adjust the surgical resection path 4312 to eliminate the need for the final staple cartridge 4353 without compromising the size of the resulting sleeve below a predetermined threshold.

[0252] In various examples, the control circuitry performing at least one aspect of the process 4300 presents a virtual firing of the proposed staple cartridge arrangement 4317, virtually separating the virtual 3D structure 4130 into a retaining portion 4318 and a removing portion 4319, as shown in FIG. 35 . The retaining portion 4318 is a virtual representation of a sleeve that results from implementing the proposed surgical resection path 4312 by firing the staple cartridge arrangement 4317. The control circuitry can further determine an estimated volume of the retaining portion 4318 and / or the removing portion 4319. The volume of the retaining portion 4318 represents the volume of the resulting sleeve. In at least one example, the volume of the retaining portion 4318 and / or the removing portion 4319 is derived from the visualization data. In another example, the volume of the retaining portion 4318 and / or the removing portion 4319 is determined from a database that stores retaining volumes, removing portion volumes, and corresponding surgical resection paths. The database can be built from previously performed surgeries on organs of the same or at least similar dimensions and resected by the same or at least similar resection path.

[0253] In various examples, a combination of predetermined average tissue thickness data based on organ situational awareness, as described in more detail above, combined with volumetric analysis from a visualization source, as well as CT, MRI, and / or ultrasound secondary imaging if available for the patient, can be utilized to select a first staple cartridge for arrangement 4317. Firing of subsequent staple cartridges in arrangement 4317 can use visualization data as well as instrument data from previous firings to optimize firing parameters. Instrument data that can be used to supplement volumetric measurements include, for example, FTF, FTC, current drawn by motors driving firing and / or closure, end effector closure gap, firing rate, tissue impedance measurements across the jaws, and / or wait or dwell time during use of the surgical instrument.

[0254] In various examples, visualization data, such as, for example, structured light data, can be used to track changes in the surface geometry of tissue being treated by a surgical instrument (e.g., surgical instrument 4600). Additionally, visualization data, such as, for example, spectral data, can be used to track important structures below the tissue surface. Structured and / or spectral data can be used to maintain contact between the configured instrument and the tissue throughout tissue treatment.

[0255] In at least one example, the end effector 4642 of the surgical instrument 4600 can be used to grasp tissue between the jaws. Once desired tissue-instrument contact is identified, for example, by user input, visualization data of the end effector and surrounding tissue associated with the desired tissue-instrument contact can be used to automatically maintain the desired tissue-surface contact throughout at least a portion of the tissue treatment. The desired tissue-surface contact can be automatically maintained, for example, by minor manipulation of the position, orientation, and / or FTC parameters of the end effector 4642.

[0256] If the surgical instrument 4600 is a handheld surgical instrument, position and / or orientation manipulations can be provided to the user in the form of instructions that can be presented, for example, on a display 4625 ( FIG. 22 ) of the surgical instrument 4600. An alert can also be generated by the surgical instrument 4600 if user manipulation is required to re-establish desired tissue-surface contact. Alternatively, non-user manipulations, such as manipulations to FTC parameters and / or joint angles, can be communicated to the controller 4620 of the surgical instrument 4600 from, for example, the surgical hub 2106 or the visualization system 2108. The controller 4620 can then cause the motor driver 4626 to implement the desired manipulation. If the surgical instrument 4600 is a surgical instrument coupled to a robotic arm of the robotic system 2110, position and / or orientation manipulations can be communicated to the robotic system 2110 from, for example, the surgical hub 2106 or the visualization system 2108.

[0257] 36A-36C , the firing of a surgical instrument 4600 is illustrated with a first staple cartridge 4652 loaded in the staple cartridge location 4317. In a first step, as shown in FIG. 36A , a first landmark 4361 and a second landmark 4362 are superimposed on the surgical resection path 4312. The landmarks 4361, 4362 are spaced apart by a distance (d1) defined by the size (e.g., 45) of the staple cartridge 4652, which represents the length of the staple line 4363 deployed by the staple cartridge 4652 on the surgical resection path 4312. The control circuitry executing one or more aspects of the process 4300 can employ visualization data, as described in more detail elsewhere herein, to superimpose the landmarks 4361, 4362 on the surgical resection path 4312 and continuously track and update their positions relative to predetermined critical structures, such as, for example, anatomical structures 4364, 4365, 4366, 4367.

[0258] 36B, during firing, the staples of staple line 4363 are deployed into the tissue and the cutting member 4645 is advanced to cut the tissue along the surgical resection path 4312 between landmarks 4361 and 4362. In various examples, as the cutting member 4645 advances, the tissue being treated is stretched and / or displaced. Stretching and / or displaced tissue beyond a predetermined threshold indicates that the cutting member 4645 is moving too quickly through the tissue being treated.

[0259] 37 is a logic flow diagram of a process 4170 illustrating a control program or logic configuration for adjusting the firing rate of a surgical instrument to account for tissue stretching and / or slippage during firing. Process 4170 includes monitoring tissue stretching / slippage during firing of the surgical instrument at 4171 and adjusting firing parameters at 4173 if tissue stretching / slippage is above a predetermined threshold at 4172.

[0260] One or more aspects of process 4170 can be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4170 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4170. Additionally or alternatively, one or more aspects of process 4170 can be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, process 4170 can be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0261] In various examples, control circuitry (e.g., block control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4170 uses visualization data from a visualization system (e.g., visualization system 100, 160, 500, 2108) to monitor 4171 tissue stretching / displacement during firing of surgical instrument 4600. In the example shown in FIG. 36B , visualization data is used to monitor 4171 tissue stretching / displacement (d) by tracking distortion of a structured light grid projected onto the tissue during firing and / or by tracking landmarks 4364, 4365, 4366, 4367 representing the positions of adjacent anatomical structures. Additionally or alternatively, tissue stretching (d) can be monitored 4171 by tracking the position of landmark 4362 during firing. In the example of FIG. 36B , the tissue stretch / displacement (d) is the difference between the distance (d1) between the landmarks 4361 and 4362 during firing and the distance (d2) between the landmarks 4361 and 4362 during firing. In either case, if the tissue stretch / displacement (d) is equal to or greater than a predetermined threshold at 4172, then the control circuitry adjusts firing parameters of the surgical instrument 4600 at 4173 to reduce the tissue stretch / displacement (d). For example, the control circuitry can cause the controller 4620 to reduce the speed of the firing motor drive assembly 4604, e.g., by reducing the draw current of the firing motor 4602 and reducing the advancement speed of the cutting member 4645. Additionally or alternatively, the control circuitry can cause the controller 4620 to pause the firing motor 4602 for a predetermined period of time to reduce the tissue stretch / displacement (d).

[0262] As shown in FIGURE 36C, after firing, the jaws of the end effector 4642 are unclamped and the stapled tissue retracts due to the fired staples in staple line 4363. FIGURE 36C shows the projected staple line length defined by a distance (d1) and the actual staple line defined by a distance (d3) that is less than distance (d1). The difference between distances d1 and d2 represents the retraction / shift distance (d').

[0263] 38 is a logic flow diagram of a process 4180 illustrating a control program or logic configuration for adjusting proposed staple cartridge placement along a proposed surgical resection path. The process 4180 includes, at 4081, monitoring retraction / shift of stapled tissue along the proposed surgical resection path after firing of the proposedly placed staple cartridge, and adjusting the proposed subsequent staple cartridge position along the proposed surgical resection path.

[0264] One or more aspects of process 4180 can be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4180 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4180. Additionally or alternatively, one or more aspects of process 4180 can be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, process 4180 can be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0265] In various examples, control circuitry (e.g., block control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4180 monitors 4181 the retraction / shift of stapled tissue along the proposed resection path 4312. In the example shown in FIG. 36C , staple line 4363 is deployed from the staple cartridge of staple cartridge arrangement 4317 into the tissue between landmarks 4361 and 4362. When the jaws of end effector 4642 are unclamped, the stapled tissue retracts / shifts by a distance (d′). Distance (d′) is the difference between the distance (d1) between landmarks 4361 and 4362 before firing, which represents the length of staple line 4361 proposed by arrangement 4317, and the distance (d3) which represents the actual length of staple line 4363.

[0266] To avoid gaps between successive staple lines, the control circuitry adjusts subsequent staple cartridge positions of the proposed arrangement 4317 along the proposed surgical resection path 4312. For example, as shown in FIG. 36C , the initially proposed staple line 4368 is removed and replaced with an updated staple line 4369 that extends through or covers the gap defined by the distance (d'). In various aspects, at 4181, the visualization data is used to monitor tissue retraction / displacement (d') by tracking distortion of a structured light grid projected onto the tissue after the jaws of the end effector 4642 are unclamped and / or by tracking landmarks 4364, 4365, 4366, 4367 that represent the positions of adjacent anatomical structures. Additionally or alternatively, at 4181, the tissue retraction distance (d') can be monitored by tracking the position of landmark 4362.

[0267] In various aspects, it may be desirable to corroborate visualized data derived from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108) with non-visualized data from a non-visualized system, or vice versa. In one example, the non-visualized system may include a ventilator that may be configured to measure non-visualized data such as the patient's lung volume, pressure, partial carbon dioxide pressure (PCO2), partial oxygen pressure (PO2), etc. Corroborating the visualized data with non-visualized data may provide a clinician with greater confidence that the visualized data derived from the visualization system is accurate. Additionally, as described in more detail below, corroborating the visualized data with non-visualized data may enable a clinician to identify postoperative complications and determine the overall efficiency of an organ. Corroboration may also be useful in cases of segmentectomy or complex lobectomy without a hiatus.

[0268] In various aspects, a clinician may need to resect a portion of a patient's organ to remove a critical structure, such as a tumor and / or other tissue. In one example, the patient's organ may be the right lung. The clinician may need to resect a portion of the patient's right lung to remove diseased tissue. However, the clinician may not want to remove too much of the patient's lung in a surgical procedure to ensure that lung function is not compromised too much. Lung function can be assessed based on the maximum lung capacity per breath, which represents maximum lung capacity. In determining how much lung can be safely removed, the clinician is limited by a predetermined maximum lung volume reduction, beyond which the lung loses its viability and complete organ resection is required.

[0269] In at least one example, lung surface area and / or volume are estimated from visualization data from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108). Lung surface area and / or volume can be estimated at maximum lung volume or maximum lung capacity per breath. In at least one example, lung surface area and / or volume can be estimated at multiple points throughout the inhalation / exhalation cycle. In at least one aspect, prior to resecting a lung portion, visualized and non-visualized data can be used to correlate the lung surface area and / or volume determined by the visualization system with the lung volume determined by the ventilator. The correlation data can be used, for example, to develop a mathematical relationship between the lung surface area and / or volume, as derived from the visualization data, and the lung volume determined by the ventilator. This relationship can be used to estimate the size of the lung portion that can be removed while maintaining maximum lung volume reduction below a predetermined threshold that maintains lung viability.

[0270] 39 illustrates a logic flow diagram of a process 4750 for suggesting surgical resection of a portion of an organ, according to at least one embodiment of the present disclosure. Process 4750 is generally performed during a surgical procedure. Process 4750 includes, at 4752, suggesting a portion of the organ to be resected based on visualization data from a surgical visualization system, where resection of the portion is configured to result in an estimated volume reduction of the organ. Process 4750 may further include, at 4754, determining a first value of a non-visualization parameter of the organ before resection of the portion, and, at 4756, determining a second value of the non-visualization parameter of the organ after resection of the portion. Furthermore, in certain examples, process 4750 may further include, at 4758, confirming the predetermined volume reduction based on the first value of the non-visualization parameter and the second value of the non-visualization parameter.

[0271] One or more aspects of process 4750 may be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4750 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4750. Additionally or alternatively, one or more aspects of process 4750 may be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, one or more aspects of process 4750 may be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0272] In various aspects, process 4750 can be implemented by a computer-implemented interactive surgical system 2100 ( FIG. 19 ) including one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104 that may include the remote server 2113. Control circuitry that performs one or more aspects of process 4750 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108).

[0273] FIG. 41A shows a set of lungs 4780 of a patient. In one embodiment, a clinician may utilize an imaging device 4782 to project 4784 a light pattern 4785, such as stripes, grid lines, and / or dots, onto the surface of the patient's right lung 4786 to enable determination of the surface topography or landscape of the patient's right lung 4786. The imaging device may be similar in various respects to imaging device 120 (FIG. 1). As described elsewhere herein, a projected light array may be employed to determine the shape defined by the surface of the patient's right lung 4786 and / or the movement of the patient's right lung 4786 during surgery. In one embodiment, the imaging device 4782 may be coupled to a structured light source 152 of the control system 133. In one embodiment, a surgical visualization system such as surgical visualization system 100 may utilize surface mapping logic 136 of the control circuitry 133 to determine the surface topography or landscape of the patient's right lung 4786, as described elsewhere herein.

[0274] A clinician can provide a surgical system, such as surgical system 2100, with the type of procedure to be performed, such as a right upper lobectomy. In addition to providing the surgical procedure to be performed, the clinician can provide the surgical system with the maximum desired volume of the organ to be removed during the surgical procedure. Based on the visualization data obtained from the imaging device 4782, the type of surgical procedure to be performed, and the maximum desired volume to be removed, the surgical system can suggest a resection path 4788 for removing a portion 4790 of the right lung that meets all of the clinician's inputs. Other methods for suggesting a surgical resection path are described elsewhere herein. To suggest the resection path 4788, the surgical system can consider any number of additional parameters.

[0275] In various examples, it may be desirable to ensure that the volume of the resected organ results in a desired volume reduction from the patient's organ. Non-visualized data from a non-visualized system can be used to confirm that the resected volume results in the desired volume reduction. In one embodiment, a ventilator can be used to measure the patient's maximum lung capacity over time.

[0276] In at least one example, a clinician may utilize the surgical system 2100 in a surgical procedure to remove a lung tumor. The control circuitry may identify the tumor from the visualization data, as described above in connection with FIGS. 13A-13E, and may suggest a surgical resection path to provide a safety margin around the tumor, as described above in connection with FIGS. 29-38. The control circuitry may further estimate the lung volume at maximum lung volume. The maximum lung volume may be measured prior to the surgical procedure using a ventilator. The control circuitry may estimate the maximum lung volume reduction associated with removing a portion of the lung, including the tumor and a safety margin of surrounding tissue, using a predetermined mathematical correlation between the visually estimated lung volume and lung volume detected by the ventilator. If the estimated lung volume reduction exceeds a predetermined safety threshold, the control circuitry may alert the clinician and / or suggest an alternative surgical resection path to reduce lung volume.

[0277] FIG. 41C shows a graph 4800 measuring a patient's maximum lung capacity over time. Prior to resection of a portion of an organ (t1), the ventilator can measure the maximum lung capacity. In FIG. 41C, at time t1, prior to resection of portion 4790, the maximum lung capacity is measured to be 6 L. In the example above, where the surgical procedure to be performed is a right upper lobectomy, the clinician may wish to remove only a volume of the patient's lung that will result in a predetermined volume reduction so as not to impair the patient's ability to breathe. In one embodiment, the clinician may wish to remove a portion that will reduce the patient's maximum lung capacity by, for example, approximately 17%. Based on the surgical procedure and the desired volume reduction, the surgical system can suggest a surgical resection path 4788 that achieves removal of the portion of the lung while maintaining the maximum lung capacity at or above 83% of the unresected maximum lung capacity.

[0278] As shown in FIG. 41C , a clinician can use ventilator data to monitor the patient's maximum lung capacity over time, such as before 4802 and after 4804 resection of lung portion 4790. At time t2, lung portion 4790 is resected along proposed resection path 4788. As a result, the maximum lung capacity measured by the ventilator decreases. The clinician can corroborate with ventilator data (maximum lung capacity before resection 4802 and maximum lung capacity after resection 4804) to ensure that the post-resection lung volume results in the desired volume reduction from the lung. As shown in FIG. 41C , after resection, the maximum lung capacity decreases to 5 L, which is approximately a 17% reduction in maximum lung capacity and roughly the same as the desired volume reduction. Using the ventilator data, the clinician can have greater confidence that the actual volume reduction is consistent with the desired volume reduction achieved by the proposed surgical resection path 4788. In other embodiments, if there is a discrepancy between the non-visualized data and the visualized data, such as a larger than expected reduction in maximum lung capacity (too much lung resection) or a smaller than expected reduction in maximum lung capacity (not enough lung resection), the clinician can determine whether appropriate action is required.

[0279] 41B, a patient's right lung 4792 is shown after removal of portion 4790. After removal of portion 4790, a clinician may inadvertently cause an air leak 4794, resulting in air leaking into the space between lung 4794 and the chest wall, causing pneumothorax 4796. As a result of air leak 4794, the patient's maximum lung capacity per breath steadily decreases over time as right lung 4792 collapses. Visualization data derived from a visualization system (e.g., visualization system 100, 160, 500, 2108) can be used to perform dynamic surface area / volume analysis of the lungs to detect air leaks by visually tracking changes in lung volume. Lung volume and / or surface area can be visually tracked at one or more points during the inspiration / expiration cycle to detect volume changes indicative of an air leak 4794. In one embodiment, as previously discussed, the projected light array from imaging device 4782 may be employed to monitor the movement of the patient's right lung 4786 over time, such as to monitor size reduction. In another embodiment, surface mapping logic, such as the surgical visualization system, surface mapping logic 136, may be utilized to determine the surface topography or landscape of the patient's right lung 4786 and monitor changes in the topography or landscape over time.

[0280] In one embodiment, a clinician can utilize a non-visualization system, such as a ventilator, to corroborate the volume reduction detected by the visualization system. Referring again to FIG. 41C , as described above, the patient's maximum lung volume can be measured at 4802 before and after removal of a portion of the lung to confirm that the desired lung volume reduction corresponds to the actual lung volume reduction. In the above example of an inadvertent air leak, at 4806, the maximum lung volume may steadily decrease over time. In one example, immediately after removal of the portion, at time t2, the clinician may notice that the maximum lung volume has decreased from 6 L to 5 L, which roughly corresponds to the desired reduction in lung volume. After removal of the portion, the surgical visualization system can monitor the patient's lung volume over time. If the surgical visualization system determines that there is a change in volume, the clinician can measure the maximum lung volume again at time t3. At time t3, the clinician may notice that the maximum lung volume has dropped from 5 L to 4 L, which confirms the data determined from the visualization system and indicates that there may be an air leak in the right lung 4792.

[0281] Additionally, the control circuitry can be configured to measure organ efficiency based on the visualized and non-visualized data. In one aspect, organ efficiency can be determined by comparing the visualized data to the difference between the non-visualized data before and after partial ablation. In one example, the visualization system can generate an ablation path that reduces maximum lung volume by 17%. The ventilator can be configured to measure the maximum lung volume before and after partial ablation. In the example shown in FIG. 41C, the maximum lung volume is reduced by approximately 17% (from 6 L to 5 L). Because the reduction in actual lung volume (17%) to desired lung volume (17%) is nearly 1:1, the clinician can determine that the lung is functionally efficient. In another example, the visualization system can generate an ablation path that reduces maximum lung volume by 17%. However, the ventilator may measure a reduction in maximum lung volume greater than 17%, such as 25%. In this example, because removing a portion of the lung results in a larger-than-expected reduction in maximum lung volume, the clinician can determine that the lung is functionally inefficient.

[0282] FIG. 40 illustrates a logic flow diagram of a process 4760 for estimating the volume reduction of an organ resulting from the removal of a selected portion of the organ, according to at least one embodiment of the present disclosure. Process 4760 is similar in many respects to the process of FIG. 4750. However, unlike process 4750, process 4760 relies on a clinician to select or suggest a surgical resection path for removing a portion of the organ during a surgical procedure. Process 4760 includes receiving input from a user, at 4762, indicating the portion of the organ to be resected. Process 4760 further includes estimating the volume reduction of the organ due to the removal of the portion, at 4764. In at least one example, the organ is a patient's lungs, and the volume reduction estimated at 4762 is a reduction in the maximum lung capacity per breath of the patient's lungs. Visualization data from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108) can be employed to estimate the volume reduction corresponding to the removal of the portion. Process 4760 may further include determining a first value of the non-visualization parameter of the organ before resection of the portion, at 4766, and determining a second value of the non-visualization parameter of the organ after resection of the portion, at 4768. Finally, process 4760 may further include confirming the estimated organ volume reduction based on the first value of the non-visualization parameter and the second value of the non-visualization parameter, at 4768.

[0283] One or more aspects of process 4760 may be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4760 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4760. Additionally or alternatively, one or more aspects of process 4760 may be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, one or more aspects of process 4760 may be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0284] In various aspects, process 4760 can be implemented by a computer-implemented interactive surgical system 2100 ( FIG. 19 ) including one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104 that may include the remote server 2113. Control circuitry that performs one or more aspects of process 4760 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108).

[0285] In one example, a clinician can provide input to a surgical visualization system, such as surgical visualization system 2100, indicating a portion of an organ to be resected. In one example, the clinician can draw a resection path on a virtual 3D structure of the organ, such as the virtual 3D structure generated in 4104 during process 4100. In another example, the visualization system can overlay a treatment layout plan, which may be in the form of a recommended treatment path, described in more detail elsewhere herein. The recommended treatment path may be based on the type of surgical procedure being performed. In one embodiment, the recommended treatment path can suggest various starting points and various resection paths from which the clinician can choose, similar to resection paths 4146, 4147, 4148 described elsewhere herein. The suggested resection paths can be determined by the visualization system to avoid certain critical structures, such as arteries. The clinician can select from the suggested resection paths until the desired resection path for removing a portion of the organ is completed.

[0286] In one example, the surgical visualization system can determine an estimated volume reduction of an organ based on a selected resection path. After resecting a predetermined portion along the resection path, a clinician may wish to use non-visualized data to confirm that the actual volume reduction matches the estimated volume reduction based on visualized data. In one embodiment, this confirmation can be performed using a procedure similar to that described above for process 4750 in which the organ is the lung. The clinician can measure the maximum lung volume of the lung before resection 4802 and after resection 4804 and compare the change in maximum lung volume to determine the actual reduction in maximum lung volume. In one example, the surgical visualization system may estimate a 17% reduction in maximum lung volume based on the clinician's proposed resection path. Before resection, the clinician may note that the maximum lung volume is 6 L (time t1). After resection, the clinician may note that the maximum lung volume is 5 L (time t2), representing approximately a 17% reduction in maximum lung volume. Using this non-visual / ventilator data, clinicians can have greater confidence that the actual volume reduction is consistent with the estimated volume reduction. In another example, if there is a discrepancy between the non-visual and visual data, such as a larger than expected reduction in peak lung volume (too much lung resection) or a smaller than expected reduction in peak lung volume (not enough lung resection), clinicians can determine if appropriate action is required.

[0287] Additionally, the control circuitry can be configured to measure organ efficiency based on the visualized and non-visualized data. In one aspect, organ efficiency can be determined by comparing the visualized data to the difference between the non-visualized data before and after the segment resection. In one example, the surgical visualization system may estimate a 17% decrease in maximum lung volume based on the clinician's desired resection path. The ventilator can be configured to measure the maximum lung volume before and after the segment resection. In the example shown in FIG. 41C, the maximum lung volume is reduced by approximately 17% (from 6 L to 5 L). Because the reduction in estimated lung volume (17%) to actual lung volume (17%) is nearly 1:1, the clinician can determine that the lungs are functionally efficient. In another example, the surgical visualization system may estimate a 17% decrease in maximum lung volume based on the clinician's desired resection path. However, the ventilator may measure a reduction in maximum lung volume greater than 17%, such as 25%. In this example, removing a portion of the lung results in a greater than expected reduction in maximum lung volume, allowing the clinician to determine that the lung is not functioning efficiently.

[0288] As described above with respect to processes 4750, 4760, a clinician can corroborate the visualized data with non-visualized data, such as by using a ventilator to measure maximum lung volume before and after removal of a portion of the lung. Another example of corroborating visualized data with non-visualized data is through capnography.

[0289] FIG. 42 shows a graph 4810 measuring the partial pressure of carbon dioxide (PCO2) exhaled by a patient over time. In another example, the partial pressure of oxygen (PO2) exhaled by a patient can be measured over time. Graph 4810 shows PCO2 measured before 4812, immediately after 4814, and one minute after 4816 the ablation. In FIG. 42, before 4812 the PCO2 is measured to be approximately 40 mmHg (at time t1). In the example above where the surgical procedure being performed is a right upper lobectomy, the visualization system may desire or estimate a 17% reduction in lung volume. PCO2 levels measured by the ventilator can be used to support this desired or estimated volume reduction.

[0290] 42, a clinician can use ventilator data to monitor a patient's PCO2 over time, such as before 4812 and after 4814 ablation of lung portion 4790. At time t2, lung portion 4790 is ablated, which can result in a drop in PCO2 measured by the ventilator at 4818. The clinician can corroborate with ventilator data (PCO2 before ablation 4812 (t1) and PCO2 after ablation 4814 (t2)) to ensure that the actual reduction in lung volume matches the estimated or desired reduction in lung volume. As can be seen in FIG. 42, immediately after ablation 4814 of portion 4790, there is a drop 4812 in PCO, which can be measured as approximately a 17% drop in PCO (approximately 33.2 mmHg). Using this non-visual / ventilator data, the clinician can have greater confidence that the actual reduction in lung volume matches the desired or estimated reduction in lung volume.

[0291] In other examples, the clinician can utilize the non-visualized / PCO2 data to determine discrepancies when compared to the visualized data. In one example, immediately after resection 4814 of portion 4790, PCO2 can be measured at 4820, which is higher than the PCO2 measured before resection 4812. The elevated PCO2 may be the result of an inadvertent bronchial obstruction during surgery, resulting in a buildup of CO2 within the patient. In another example, immediately after resection 4814 of portion 4790, PCO2 can be measured at 4822, which is lower than the PCO2 measured before resection 4812 and may be measured lower than expected. The decreased PCO2 may be the result of an inadvertent vascular obstruction during surgery, resulting in less O2 being delivered to the body and, consequently, less CO2 being produced. In either case, the clinician can take appropriate action to remedy the situation.

[0292] Additionally, the change in PCO2 can be measured at a time other than immediately after the ablation 4814, such as one minute after the ablation 4816 (e.g., at time t3). At time t3, other bodily functions (e.g., the liver) compensate for the change in PCO2 as a result of the ablation. In this situation, PCO2 can be measured at approximately 40 mmHg, i.e., roughly the same as before the ablation 4812. A difference measured at time t3 between the PCO2 measured before the ablation 4812 may indicate an inadvertent occlusion, as discussed above. For example, at time t3, the PCO2 may be measured higher at 4824 than before the ablation 4812, indicating a possible inadvertent bronchial obstruction, or the PCO2 may be measured lower at 4826 than before the ablation 4812, indicating a possible inadvertent vascular obstruction.

[0293] Additionally, the control circuitry can be configured to measure organ efficiency based on the visualized and non-visualized data. In one aspect, organ efficiency can be determined by comparing the visualized data to the difference between the non-visualized data before and after the resection of a portion. In one example, the surgical visualization system may estimate a 17% decrease in lung volume based on the clinician's desired resection path. The ventilator may be configured to measure PCO2 before and after the resection of a portion. In the embodiment shown in FIG. 42, immediately after the resection 4814, PCO2 decreases by approximately 17%. Because the (17%) decrease in PCO2 to the estimated (17%) decrease in lung volume is nearly 1:1, the clinician can determine that the lungs are functionally efficient. In another example, the surgical visualization system may estimate a 17% decrease in lung volume based on the clinician's desired resection path. However, the ventilator may measure a decrease in PCO2 greater than 17%, such as 25%. In this example, removing a portion of the lung results in a larger-than-expected decrease in PCO2, allowing the clinician to determine that the lungs are functionally inefficient.

[0294] In addition to the above-mentioned maximum lung capacity and maximum PCO2, non-visualized parameters including blood pressure or EKG data can be utilized. EKG data provides approximate frequency data regarding arterial deformation. This frequency data, which shows changes in surface geometry within similar frequency ranges, can help identify important vascular structures.

[0295] As mentioned above, it may be desirable to utilize non-visualized data from a non-visualized system to corroborate visualized data derived from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108). In the above example, the non-visualized data is a means to corroborate the visualized data after a portion of the organ has already been resected. In this example, it may be desirable to supplement the visualized data with non-visualized data before the portion of the organ is resected. In one example, the non-visualized data can be used in conjunction with the visualized data to help determine characteristics of the organ undergoing surgery. In one aspect, this characteristic may be an abnormality in the organ tissue that may not be suitable for cutting. The non-visualized and visualized data may help inform the surgical visualization system and clinician about areas to avoid when planning the resection path of the organ. This may also be useful in the case of a segmentectomy or a complex lobectomy without a hiatus.

[0296] 43 shows a logic flow diagram of a process 4850 for detecting a tissue abnormality based on visualized data and non-visualized data in accordance with at least one aspect of the present disclosure. Process 4850 is generally performed during a surgical procedure. Process 4850 may include receiving, at 4852, first visualization data of the organ at a first state from a surgical visualization system and determining, at 4854, a first value of a non-visualized parameter for the organ at the first state. Process 4850 may further include receiving, at 4856, second visualization data of the organ at a second state from the surgical visualization system and determining, at 4858, a second value of the non-visualized parameter for the organ at the second state. The process may also include detecting the tissue abnormality based on the first visualization data, the second visualization data, the first value of the non-visualized parameter, and the second value of the non-visualized parameter at 4860.

[0297] One or more aspects of process 4850 may be performed by one or more of the control circuits described by this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4850 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A) that includes a processor and a memory that stores a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4850. Additionally or alternatively, one or more aspects of process 4850 may be performed by a combinational logic circuit (e.g., control circuit 410 of FIG. 2B) and / or a sequential logic circuit (e.g., control circuit 420 of FIG. 2C). Furthermore, one or more aspects of process 4850 may be performed by any suitable circuit having any suitable hardware and / or software components that may be located in or associated with various suitable systems described by this disclosure.

[0298] In various aspects, process 4850 can be implemented by a computer-implemented interactive surgical system 2100 ( FIG. 19 ) that includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with the cloud 2104 that may include the remote server 2113. Control circuitry that performs one or more aspects of process 4850 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108).

[0299] FIG. 44A illustrates a patient's right lung 4870 in a first state 4862. In one example, the first state 4862 can be a deflated state. In another example, the first state 4862 can be a collapsed state. An imaging device 4872 is shown inserted through a cavity 4874 in the patient's chest wall 4876. A clinician can utilize the imaging device 4872 to project a light pattern 4882, such as stripes, grid lines, and / or dots, onto the surface of the patient's right lung 4870 to enable determination of the topography or landscape of the surface of the patient's right lung 4870 at 4880. The imaging device can be similar in various respects to imaging device 120 ( FIG. 1 ). As described elsewhere herein, a projected light array can be employed to determine the shape defined by the surface of the patient's right lung 4870 and / or the movement of the patient's right lung 4870 during surgery. In one embodiment, the imaging device 4782 can be coupled to a structured light source 152 of the control system 133. In one embodiment, as described elsewhere herein, a surgical visualization system such as surgical visualization system 100 may utilize surface mapping logic 136 of control circuitry 133 to determine the surface topography or landscape of the patient's right lung 4786. In a first state 4862 of the right lung 4870, a ventilator may be used to measure a parameter of the right lung 4870, such as first state pressure (P1, or positive end-expiratory pressure (PEEP)), or first state volume (V1).

[0300] 44B shows the patient's right lung 4870 in a second state 4864. In one example, the second state 4864 can be a partially inflated state. In another example, the second state 4864 can be a fully inflated state. The imaging device 4872 can be configured to continuously emit a light pattern 4882 onto the surface of the lung 4870 to enable determination of the surface topography or landscape of the patient's right lung 4870 in the second state 4864, at 4880. In the second state 4864 of the right lung 4870, a ventilator can be used to measure parameters of the right lung 4870, such as a second state pressure (P2) that is greater than the pressure P1 of the first state 4862, and a second state volume (V2) that is greater than the volume V1 of the first state 4862.

[0301] Based on the surface topography determined from the surgical visualization system and the imaging device 4872, along with non-visualized data (pressure / volume) determined from the ventilator, the surgical visualization system can be configured to determine tissue abnormalities in the right lung 4870. In one example, in a first state 4862, the imaging device 4872 can determine a topography of the right lung 4870 in a first state 4662 (shown in FIG. 44A and in more detail in FIG. 44C ), and the ventilator can determine the pressure / volume at the first state. In a second state 4864, the imaging device 4872 can determine a topography of the right lung 4870 in a second state 4864 (shown in FIG. 44D and in more detail in FIG. 44D ), and the ventilator can determine a pressure / volume at the second state that is greater than the pressure / volume at the first state due to the lung being partially or fully inflated. The visualization system can be configured to monitor changes in the topography of the right lung 4870 based on known increases in pressure / volume. In one embodiment, this pressure / volume measurement from the ventilator can be correlated with the surface deformation of the right lung 4870 to identify diseased areas within the lung and aid in stapler placement.

[0302] 44B and 44D, when pressure increases from P1 to P2 (volume increases from V1 to V2), the surface topography determined from structured light 4880 changes compared to first state 4862. In one example, light pattern 4882 can be dots, with the dots spaced a distance apart as lung size increases. In another example, light pattern 4882 can be grid lines, with the grid lines spaced apart or contoured as lung size increases. Based on the known pressure increase, the imaging device can determine regions 4886 that did not change following the known pressure and volume increases. For example, if imaging device 4872 emits a pattern 4882 of grid lines and dots onto the surface of right lung 4870 (as shown in FIGS. 6A-6D), the visualization system can be configured to monitor the contours of the grid lines and the relative positions of the dots with respect to the known pressure / volume increase. If the visualization system notices irregularities in the spacing of the dots or the position and curvature of the grid lines, the visualization system can determine that these areas correspond to possible tissue abnormalities such as subsurface voids 4886 or areas where critical structures such as tumors 4884 may be located. In one embodiment, referring to process 4100 identifying anatomical structures of at least a portion 4105 of an anatomical organ relevant to a surgical procedure, process 4100 can identify abnormalities as described herein above and overlay these abnormalities on the 3D structure.

[0303] In one example, a patient may have emphysema, a lung disease that causes shortness of breath. In people with emphysema, the air sacs (alveoli) in the lungs are damaged, and over time, the lining of the air sacs weakens and ruptures, creating large air spaces instead of many small ones. This reduces the internal surface area of ​​the lungs available for O2 / CO2 exchange and therefore reduces the amount of oxygen reaching the bloodstream. In addition, damaged alveoli do not function properly, trapping stale air and leaving no room for fresh, oxygen-rich air. Because air spaces within the lungs of emphysema patients represent areas of reduced tissue thickness, stapling outcomes in those areas may be affected. The tissue also weakens, resulting in ruptured alveoli and poor ability to hold staples through them.

[0304] As emphysematous lungs expand and contract, areas with subsurface voids deform differently due to pressure changes compared to healthy tissue. Using the process 4850 described above, weak tissue areas containing these subsurface voids can be detected, informing the clinician to avoid stapling through these areas, thereby reducing the likelihood of postoperative air leaks. The tissue deformation forces of this process 4850 can detect these differences, allowing the surgeon to guide stapler placement.

[0305] In a second example, a patient may have cancer. Prior to the procedure, the tumor may have been irradiated, damaging not only the tissue but also the surrounding tissue. Radiation often changes the properties of the tissue, making it stiffer and less compressible. If the surgeon needs to staple across this tissue, they should consider the change in tissue stiffness when selecting the staple reload type (e.g., stiffer tissue will require a taller formed staple).

[0306] As the lung expands and contracts, areas with stiffer tissue will deform differently compared to healthy tissue because the lung is less flexible. The tissue deformation forces of this process 4850 can detect these differences, allowing the surgeon to guide stapler placement as well as cartridge / reload color selection.

[0307] In another aspect, a memory, such as memory 134, can be configured to store lung surface topographies for known pressures and volumes. In this example, an imaging device, such as imaging device 4872, can emit a light pattern to determine the topography of the patient's lung surface at a first known pressure or volume. A surgical system, such as surgical system 2100, can be configured to compare a first determined topography at a known first pressure or volume to a topography stored in memory 134 for a given first pressure or volume. Based on this comparison, the visualization system can be configured to indicate possible tissue abnormalities in only a single condition. The visualization system can focus on these possible abnormality regions and proceed to determine the patient's lung surface topography at a second known pressure or volume. The visualization system can compare the second determined surface topography to the topography stored in memory for a given second pressure or volume, as well as the topography determined at the first known pressure or volume. If the visualization system determines a region of possible abnormality that overlaps with a first determined region of possible abnormality, the visualization system can be configured to indicate the overlapping region as a possible abnormality with greater confidence based on a comparison at known first and second pressures or volumes.

[0308] In addition to the above, PO2 measurements from the ventilator can compare inflated lung volumes, such as V2, to deflated lung volumes, such as V1. Volumetric comparisons utilize EKG data to compare inhaled and exhaled airflow, which can be compared to blood oxygen levels. This can also be compared to anesthesia gas exchange measurements to determine respiratory volume versus oxygen uptake versus sedation. Additionally, EKG data provides approximate frequency data regarding arterial deformation. This frequency data, which shows changes in surface geometry within similar frequency ranges, can help identify important vascular structures.

[0309] In another embodiment, current tracking / treatment information can be compared to a preoperative planning simulation. For difficult or high-risk procedures, clinicians can utilize preoperative patient scans to simulate the surgical approach. This data set can be compared against real-time measurements on a display, such as display 146, to help enable surgeons to follow a specific preoperative plan based on training runs. This requires the ability to match fiducial landmarks between the preoperative scan / simulation and the current visualization. Some methods simply use boundary tracking of the object. Insights into how the current device-tissue interaction compares to previous interactions (per patient) or predicted interactions (database or previous patients) for tissue type differentiation, relative tissue deformation assessment, or subsurface structural differences can be stored in a memory, such as memory 134.

[0310] In one embodiment, the surface geometry may be a function of instrument position. If the change in surface geometry is not measured for each change in instrument position, a surface reference can be selected. As the instrument interacts with tissue and deforms the surface geometry, the change in surface geometry as a function of instrument position can be calculated by the surgical system. For a given change in instrument position when contacting the tissue, the change in tissue geometry may be different in areas containing subsurface structures, such as critical structures 4884, than in areas not containing such structures, such as subsurface voids 4886. In one example, such as thoracic surgery, this may be above the airway as opposed to only in parenchymal tissue. A running average of the change in instrument position and the change in surface geometry can be calculated for a given patient by the surgical system using the surgical visualization system, tailored to given patient-specific variations, or the value can be compared to a second set of previously collected data.

[0311] Example Clinical Uses The various surgical visualization systems disclosed herein may be employed in one or more of the following clinical applications: The following clinical applications are non-exhaustive and merely exemplary applications for one or more of the various surgical visualization systems disclosed herein.

[0312] The surgical visualization system, as disclosed herein, can be employed in many different types of procedures in different specialties, such as, for example, urology, gynecology, oncology, colorectal surgery, thoracic surgery, bariatric / gastrocology, and hepatobiliary-pancreatic surgery (HPB). For example, in urological surgery, such as a prostatectomy, the urinary tract may be detected within fat or connective tissue, and / or nerves may be detected within fat. For example, in gynecological oncology surgery, such as a hysterectomy, and colorectal surgery, such as a low anterior resection (LAR), the ureter may be detected within fat and / or connective tissue. For example, in thoracic surgery, such as a lobectomy, blood vessels may be detected within lung or connective tissue, and / or nerves may be detected within connective tissue (e.g., esophagostomy). For bariatric surgery, blood vessels may be detected within fat. For example, in HPB procedures such as hepatectomy or pancreatectomy, blood vessels may be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts may be detected in parenchymal tissue (liver or pancreas).

[0313] In one example, a clinician may wish to remove an endometrial fibroid. From a preoperative magnetic resonance imaging (MRI) scan, the clinician may know that the endometrial fibroid is located on the surface of the intestine. Therefore, the clinician may wish to know during surgery which tissue constitutes part of the intestine and which tissue constitutes part of the rectum. In such an example, a surgical visualization system, as disclosed herein, can indicate the different types of tissue (intestine vs. rectum) and communicate that information to the clinician via the imaging system. Furthermore, the imaging system can determine and communicate the proximity of a surgical device to the selected tissue. In such an example, the surgical visualization system can improve the efficiency of the procedure without significant complications.

[0314] In another example, a clinician (e.g., a gynecologist) may stay away from certain anatomical areas to avoid getting too close to critical structures, so the clinician may not remove all of the endometriosis, for example. A surgical visualization system, as disclosed herein, can enable the gynecologist to mitigate the risk of getting too close to critical structures, allowing the surgical device to get close enough to remove all of the endometriosis, improving patient outcomes (democratizing surgery). Such a system can allow the surgeon to "keep moving" during surgery, instead of repeatedly stopping and restarting to identify areas to avoid, for example, during the application of therapeutic energy, particularly ultrasound or electrosurgical energy. In gynecological applications, the uterine arteries and ureters are important critical structures, and the system may be particularly useful for hysterectomies and endometrial surgeries, given the presentation and / or thickness of the tissues involved.

[0315] In another example, a clinician may risk dissecting a vessel too close and in a location that could affect the blood supply to lobes other than the targeted lobe. Furthermore, anatomical differences between patients may result in dissecting a vessel (e.g., a branch vessel) that affects different lobes based on the particular patient. A surgical visualization system, as disclosed herein, can enable identification of the correct vessel at the desired location, thereby enabling the clinician to ensure that the appropriate anatomy is dissected. For example, the system can confirm that the correct vessel is in the correct location, allowing the clinician to then safely divide the vessel.

[0316] In another example, due to uncertainty about vascular anatomy, a clinician may make multiple incisions before finding the best location. However, since more incisions may increase the risk of bleeding, it is desirable to find the best location the first time. A surgical visualization system, as disclosed herein, can minimize the number of incisions by indicating the correct vessels and the best locations for incisions. For example, the ureter and cardinal ligaments are densely packed and present unique challenges during dissection. In such instances, minimizing the number of incisions may be particularly desirable.

[0317] In another example, a clinician (e.g., an oncological surgeon) removing cancerous tissue may desire to know the identification of critical structures, the localization of the cancer, the staging of the cancer, and / or an assessment of the health of the tissue. Such information goes beyond what the clinician sees with the "naked eye." A surgical visualization system, as disclosed herein, can determine such information and / or communicate such information to the clinician during surgery to enhance intraoperative decision-making and improve surgical outcomes. In certain examples, the surgical visualization system may be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, for example, using either an endoscope or an exoscope.

[0318] In another example, a clinician (e.g., an oncology surgeon) may want to turn off one or more warnings regarding the proximity of a surgical tool to one or more critical structures to avoid being too conservative during a surgical procedure. In another example, a clinician may want to receive certain types of warnings, such as haptic feedback (e.g., vibration / buzzer) to indicate proximity and / or a “no-fly zone” to remain sufficiently far away from one or more critical structures. A surgical visualization system, as disclosed herein, can provide adaptability based, for example, on the clinician's experience and / or the desired aggressiveness of the procedure. In such an example, the system provides a balance between “knowing too much” and “knowing enough” to anticipate and avoid critical structures. A surgical visualization system can assist in planning the next step(s) during a surgical procedure.

[0319] Various aspects of the subject matter described herein are illustrated in the following numbered examples. Example 1. A surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ, the surgical system including at least one imaging device; and control circuitry configured to: identify anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device; propose a surgical resection path for removing a portion of the anatomical organ with the surgical instrument, the surgical resection path being determined based on the anatomical structures; and present parameters of the surgical instrument in accordance with the surgical resection path. Example 2. The surgical system of Example 1, wherein the parameters are presented along the surgical resection path. Example 3. The surgical system of Example 1 or 2, wherein the surgical resection path is superimposed on a 3D structure of at least a portion of the anatomical organ. Example 4. The surgical system of example 3, wherein the parameters are superimposed along the surgical resection path. Example 5. A surgical system described in any one of Examples 1 to 4, wherein the surgical instrument is a surgical stapler, and the parameters include at least one of staple cartridge size, staple cartridge color, staple cartridge type, and staple cartridge length. Example 6. A surgical system described in any one of Examples 1 to 5, wherein the parameters of the surgical instrument are functional parameters including at least one of closure force (FTC), firing force (FTF), firing velocity, and closure velocity parameters. Example 7. The surgical system of any one of Examples 1-6, wherein the parameters are selected according to at least one tissue thickness along the surgical resection path. Example 8. The surgical system of any one of Examples 1-7, wherein the control circuitry is further configured to superimpose tissue parameters along the surgical resection path. Example 9. The surgical system of Example 8, wherein the tissue parameters include at least one of tissue thickness, tissue type, and volumetric outcome of the surgical resection path. Example 10. A surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ, the surgical system including at least one imaging device; and control circuitry configured to: identify anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device; propose a surgical resection path for removing a portion of the anatomical organ with the surgical instrument, the surgical resection path being determined based on the anatomical structures; and adjust parameters of the surgical instrument in accordance with the surgical resection path. Example 11. The surgical system of Example 10, wherein the control circuit is configured to adjust the parameters of the surgical instrument according to tissue parameters along the surgical resection path. Example 12. The surgical system of Example 10 or 11, wherein the surgical resection path is superimposed on a 3D structure of at least a portion of the anatomical organ. Example 13. A surgical system described in any one of Examples 10 to 12, wherein the parameters of the surgical instrument are functional parameters including at least one of closure force (FTC), firing force (FTF), firing velocity, and closure velocity parameters. Example 14. A surgical system according to any one of Examples 10 to 13, wherein the parameters are selected according to at least one tissue thickness along the surgical resection path. Example 15. A surgical system according to any one of Examples 10 to 14, wherein the control circuitry is further configured to superimpose tissue parameters along the surgical resection path. Example 16. The surgical system of Example 15, wherein the tissue parameters include at least one of tissue thickness, tissue type, and volumetric outcome of the surgical resection path. Example 17. A surgical system for use with a surgical stapling instrument in a surgical procedure performed on an anatomical organ, the surgical system including at least one imaging device and a control circuit configured to: propose a surgical resection path for removing a portion of the anatomical organ with the surgical stapling instrument; propose a staple cartridge placement along the surgical resection path; and monitor tissue displacement along the proposed surgical resection ...

Claims

1. 1. A surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ, the surgical system comprising: at least one imaging device; A control circuit comprising: identifying anatomical structures relevant to the surgical procedure from visualization data from the at least one imaging device; proposing a surgical resection path for removing a portion of the anatomical organ with the surgical instrument, the surgical resection path being determined based on the anatomical structure; adjusting parameters of the surgical instrument according to the surgical resection path; a control circuit configured to: A surgical system comprising:

2. The surgical system of claim 1 , wherein the control circuitry is configured to adjust the parameters of the surgical instrument according to tissue parameters along the surgical resection path.

3. The surgical system of claim 1 , wherein the surgical resection path is superimposed on a 3D structure of at least a portion of the anatomical organ.

4. The surgical system of claim 1 , wherein the parameter of the surgical instrument is a functional parameter including at least one of a closure force (FTC), a firing force (FTF), a firing rate, and a closure rate parameter.

5. The surgical system of claim 1 , wherein the parameters are selected according to at least one tissue thickness along the surgical resection path.

6. The surgical system of claim 1 , wherein the control circuitry is further configured to superimpose tissue parameters along the surgical resection path.

7. The surgical system of claim 6 , wherein the tissue parameters include at least one of tissue thickness, tissue type, and volumetric outcome of the surgical resection path.

8. 1. A surgical system for use with a surgical stapling instrument in a surgical procedure performed on an anatomical organ, the surgical system comprising: at least one imaging device; A control circuit comprising: proposing a surgical resection path for removing a portion of the anatomical organ with the surgical stapling instrument; Proposing staple cartridge placement along the surgical resection path; monitoring tissue displacement along the proposed surgical resection path due to deployment of a staple line from the staple cartridge into tissue of the staple cartridge deployment; a control circuit configured as follows: A surgical system comprising:

9. 9. The surgical system of claim 8, wherein the deviation is detected during firing of the surgical stapling instrument, and the control circuitry is further configured to adjust a parameter of the surgical stapling instrument to reduce the deviation.

10. The surgical system of claim 9 , wherein the control circuitry is configured to regulate a firing rate of the surgical stapling instrument.

11. 9. The surgical system of claim 8, wherein the deviation is detected after the jaws of the surgical stapling instrument are unclamped following deployment of a staple line from the staple cartridge of the staple cartridge arrangement into the tissue, and the control circuitry is further configured to adjust a position of a next staple cartridge of the staple cartridge arrangement to compensate for the deviation.

Images