Surgical system for superimposing surgical instrument data onto a virtual 3D construct of an organ

Abstract

The present invention discloses a surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical structure relevant to the surgical procedure based on visualization data from the at least one imaging device, recommend a surgical resection path for removing a portion of the anatomical organ using the surgical instrument, and present parameters of the surgical instrument based on the surgical resection path. The surgical resection path is determined based on the anatomical structure.

Description

Background Art

[0001] Surgical systems are often combined with imaging systems that allow clinicians to view the surgical site and / or one or more portions thereof, for example, on one or more displays (such as monitors). The displays can be local to the operating room and / or remote. The imaging system may include a scope with a camera that views the surgical site and transmits the view to a display that the clinician can view. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-nephroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. Imaging systems may be limited by the information they can identify and / or convey to clinicians. For example, some imaging systems may not be able to identify certain hidden structures, physical contours, and / or dimensions in three-dimensional space during surgery. Additionally, some imaging systems may not be able to transmit and / or convey certain information to clinicians during surgery. Summary of the Invention

[0002] In one general aspect, a surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ is disclosed. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical structure relevant to the surgical procedure based on visualization data from the at least one imaging device, recommend a surgical resection path for removing a portion of the anatomical organ with the surgical instrument, and present parameters of the surgical instrument based on the surgical resection path. The surgical resection path is determined based on the anatomical structure.

[0003] In another general aspect, a surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ is disclosed. The surgical system includes at least one imaging device and control circuitry configured to identify an anatomical structure relevant to the surgical procedure based on visualization data from the at least one imaging device, recommend a surgical resection path for removing a portion of the anatomical organ with the surgical instrument, and adjust parameters of the surgical instrument based on the surgical resection path. The surgical resection path is determined based on the anatomical structure.

[0004] In yet another general aspect, a surgical system for use with a surgical stapling instrument during a surgical procedure performed on an anatomical organ is disclosed. The surgical system includes at least one imaging device and control circuitry configured to recommend a surgical resection path for removing a portion of the anatomical organ with the surgical stapling instrument, recommend a staple cartridge arrangement along the surgical resection path, and monitor displacement of tissue along the recommended surgical resection path due to deployment of staple lines from a staple cartridge of the staple cartridge arrangement into the tissue. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The novel features of the various aspects are set forth with particularity in the appended claims. However, the described aspects, both as to organization and method of operation, may be best understood by reference to the following description taken in conjunction with the accompanying drawings, in which:

[0006] Figure 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, the surgical visualization system configured to identify critical structures beneath a tissue surface, according to at least one aspect of the present disclosure.

[0007] Figure 2 is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0008] Figure 2A Control circuitry configured to control various aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0009] Figure 2B Combinatorial logic circuitry configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure is shown.

[0010] Figure 2C A sequential logic circuit configured to control various aspects of a surgical visualization system according to at least one aspect of the present disclosure is shown.

[0011] Figure 3 is a description according to at least one aspect of the present disclosure Figure 1 Triangulation between surgical devices, imaging devices and critical structures to determine the depth of critical structures below the tissue surface a Schematic diagram of .

[0012] Figure 4 is a schematic diagram of a surgical visualization system configured to identify a critical structure below a tissue surface according to at least one aspect of the present disclosure, wherein the surgical visualization system includes a method for determining a depth d of the critical structure below the tissue surface. a Pulsed light source.

[0013] Figure 5 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, the surgical visualization system configured to identify critical structures beneath a tissue surface, according to at least one aspect of the present disclosure.

[0014] Figure 6 is a schematic diagram of a surgical visualization system including a three-dimensional camera according to at least one aspect of the present disclosure, wherein the surgical visualization system is configured to identify critical structures embedded within tissue.

[0015] Figure 7A and Figure 7B According to at least one aspect of the present disclosure Figure 6 A 3D camera captures a view of the key structure, where Figure 7A is the view from the left lens of the 3D camera, and Figure 7B is the view from the right lens of the 3D camera.

[0016] Figure 8 According to at least one aspect of the present disclosure Figure 6 Schematic diagram of a surgical visualization system in which a camera-key structure distance d from a three-dimensional camera to a key structure can be determined w .

[0017] Figure 9 is a schematic diagram of a surgical visualization system utilizing two cameras to determine the position of an embedded critical structure according to at least one aspect of the present disclosure.

[0018] Figure 10A is a schematic diagram of a surgical visualization system utilizing a camera that is axially moved between a plurality of known positions to determine the position of an embedded critical structure in accordance with at least one aspect of the present disclosure.

[0019] Figure 10B According to at least one aspect of the present disclosure Figure 10A Schematic diagram of a surgical visualization system in which a camera is moved axially and rotationally between multiple known positions to determine the position of embedded critical structures.

[0020] Figure 11 is a schematic diagram of a control system for a surgical visualization system according to at least one aspect of the present disclosure.

[0021] Figure 12 is a schematic diagram of a structured light source for a surgical visualization system according to at least one aspect of the present disclosure.

[0022] Figure 13A is a graph of absorption coefficients at different wavelengths for various biological materials according to at least one aspect of the present disclosure.

[0023] Figure 13B is a schematic diagram of visualizing an anatomical structure via a spectral surgical visualization system according to at least one aspect of the present disclosure.

[0024] Figures 13C to 13E Depicted are exemplary hyperspectral recognition features for distinguishing anatomical structures from obscurants in accordance with at least one aspect of the present disclosure, wherein Figure 13C is a graphic representation of the ureteral features and obscurations, Figure 13D is a graphical representation of arterial features and obscurations, and Figure 13E is a graphical representation of neural signatures and masking.

[0025] Figure 14 is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense distances to critical anatomical structures according to at least one aspect of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) positioned on a common device.

[0026] Figure 15 According to at least one aspect of the present disclosure Figure 14 Schematic diagram of the transmitted wave, received wave, and the delay between the transmitted and received waves of the NIR time-of-flight measurement system.

[0027] Figure 16 An NIR time-of-flight measurement system configured to sense distances to different structures, including a transmitter (emitter) and a receiver (sensor) on separate devices, is shown in accordance with at least one aspect of the present disclosure.

[0028] Figure 17 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure.

[0029] Figure 18 A surgical system for performing a surgical procedure in an operating room according to at least one aspect of the present disclosure.

[0030] Figure 19 A computer-implemented interactive surgical system according to at least one aspect of the present disclosure is shown.

[0031] Figure 20 A diagram illustrating a situational awareness surgical system in accordance with at least one aspect of the present disclosure is shown.

[0032] Figure 21 Shown is a timeline depicting a hub's situational awareness in accordance with at least one aspect of the present disclosure.

[0033] Figure 22 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control procedure or logic configuration for associating visualization data with instrumentation data.

[0034] Figure 23 is a schematic diagram of a surgical instrument according to at least one aspect of the present disclosure.

[0035] Figure 24 is a graph depicting a composite data set and force to close (“FTC”) and force to fire (“FTF”) virtual gauges according to at least one aspect of the present disclosure.

[0036] Figure 25A A general view of a screen of a visualization system showing a real-time feed of an end effector in a surgical field of view of a surgical procedure is shown in accordance with at least one aspect of the present disclosure.

[0037] Figure 25B An enhanced view of a screen of a visualization system showing a real-time feed of an end effector in a surgical field of view of a surgical procedure is shown in accordance with at least one aspect of the present disclosure.

[0038] Figure 26 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control program or logic configuration for synchronizing movement of a virtual representation of an end effector component with actual movement of the end effector component.

[0039] Figure 27 Anatomical structures in a body wall and a cavity beneath the body wall are shown, wherein a trocar is passed through the body wall into the cavity, and the screen displays the distance of the trocar from the anatomical structure, the risks associated with presenting a surgical instrument through the trocar, and the estimated operation time associated therewith, in accordance with at least one aspect of the present disclosure.

[0040] Figure 28 A virtual three-dimensional ("3D") configuration of a stomach exposed to structured light from a structured light projector is shown in accordance with at least one aspect of the present disclosure.

[0041] Figure 29 is a logic flow diagram of a process according to at least one aspect of the present disclosure depicting a control procedure or logic configuration for associating visualization data with instrumentation data, wherein blocks with dashed lines represent alternative implementations of the process.

[0042] Figure 30 Shown is a virtual 3D configuration of a stomach exposed to structured light from a structured light projector in accordance with at least one aspect of the present disclosure.

[0043] Figure 31 is a logic flow diagram of a process according to at least one aspect of the present disclosure depicting a control program or logic configuration for recommending a resection path for removing a portion of an anatomical organ, wherein blocks with dashed lines represent alternative implementations of the process.

[0044] Figure 32A Shown is a real-time view of a surgical field on a screen of a visualization system at the start of a surgical procedure in accordance with at least one aspect of the present disclosure.

[0045] Figure 32B According to at least one aspect of the present disclosure Figure 32AA magnified view of a portion of a surgical field outlining the recommended surgical resection path superimposed on the surgical field.

[0046] Figure 32C According to at least one aspect of the present disclosure, forty-three minutes after the start of the surgical procedure Figure 32B Live view of the surgical field.

[0047] Figure 32D According to at least one aspect of the present disclosure Figure 32C A magnified view of the surgical field outlines the modifications to the recommended surgical resection path.

[0048] Figure 33 is a logic flow diagram of a process according to at least one aspect of the present disclosure, which depicts a control program or logic configuration for presenting parameters of a surgical instrument onto or near a recommended surgical resection path, wherein blocks with dashed lines represent alternative implementations of the process.

[0049] Figure 34 A virtual 3D configuration of the stomach of a patient undergoing sleeve gastrectomy according to at least one aspect of the present disclosure is shown.

[0050] Figure 35 Shown Figure 34 Complete virtual resection of the stomach.

[0051] Figures 36A to 36C Firing of a surgical stapling instrument according to at least one aspect of the present disclosure is illustrated.

[0052] Figure 37 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control program or logic configuration for adjusting the firing speed of a surgical instrument.

[0053] Figure 38 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control program or logic configuration for a recommended staple cartridge placement along a recommended surgical resection path.

[0054] Figure 39 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control program or logic configuration for recommending surgical resection of an organ portion.

[0055] Figure 40 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control program or logic configuration for estimating an amount of organ volume reduction resulting from removal of a selected portion of an organ.

[0056] Figure 41A A patient's lung, including a portion to be resected during surgery, is shown exposed to structured light in accordance with at least one aspect of the present disclosure.

[0057] Figure 41B Shows the portion after cutting out according to at least one aspect of the present disclosure. Figure 41A of the patient's lungs.

[0058] Figure 41C ] shows measurements from a lung before and after resection of the portion of the lung according to at least one aspect of the present disclosure. Figure 41A and Figure 41B A graph of the peak lung capacity of a patient's lungs.

[0059] Figure 42 Graphs showing measurements of partial pressure of carbon dioxide ("PCO2") in a patient's lungs before, immediately after, and one minute after resection of a portion of the lung in accordance with at least one aspect of the present disclosure are shown.

[0060] Figure 43 is a logic flow diagram of a process according to at least one aspect of the present disclosure that depicts a control program or logic configuration for detecting tissue anomalies using visual and non-visual data.

[0061] Figure 44A The right lung is shown in a first state with an imaging device emitting a pattern of light onto a surface thereof, in accordance with at least one aspect of the present disclosure.

[0062] Figure 44B The second state of the embodiment of the present invention is shown in accordance with at least one aspect of the present disclosure. Figure 44A The right lung, where an imaging device transmits patterns of light onto its surface.

[0063] Figure 44C According to at least one aspect of the present disclosure Figure 44A The top part of the right lung.

[0064] Figure 44D According to at least one aspect of the present disclosure Figure 44B The top part of the right lung. DETAILED DESCRIPTION

[0065] The applicant of the present application owns the following U.S. patent applications, each of which is incorporated herein by reference in its entirety:

[0066] Attorney Docket No. END9228USNP1 / 190580-1M, entitled “METHOD OF USING IMAGING DEVICES IN SURGERY”;

[0067] Attorney Docket No. END9227USNP1 / 190579-1, entitled “ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM”;

[0068] Attorney Docket No. END9226USNP1 / 190578-1, titled “SURGICAL SYSTEM CONTROLBASED ON MULTIPLE SENSED PARAMETERS”;

[0069] Attorney Docket No. END9225USNP1 / 190577-1, entitled “ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS”;

[0070] Attorney Docket No. END9224USNP1 / 190576-1, titled “ADAPTIVE SURGICAL SYSTEMCONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS”;

[0071] Attorney Docket No. END9223USNP1 / 190575-1, entitled “SURGICAL SYSTEMSCORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA”;

[0072] Attorney Docket No. END9222USNP1 / 190574-1, entitled “SURGICAL SYSTEMS FORGENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED”;

[0073] Attorney Docket No. END9220USNP1 / 190572-1, entitled “SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS”;

[0074] Attorney Docket No. END9219USNP1 / 190571-1, entitled “SYSTEM AND METHOD FORDETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE”;

[0075] Attorney Docket No. END9218USNP1 / 190570-1, titled “VISUALIZATION SYSTEMSUSING STRUCTURED LIGHT”;

[0076] Attorney Docket No. END9217USNP1 / 190569-1, entitled “DYNAMIC SURGICALVISUALIZATION SYSTEMS”; and

[0077] Attorney Docket No. END9216USNP1 / 190568-1, titled “ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM.”

[0078] The applicant of the present application owns the following U.S. patent applications filed on March 15, 2019, each of which is incorporated herein by reference in its entirety:

[0079] U.S. patent application serial number 16 / 354,417, entitled “Input Controls for Robotics Urgery”;

[0080] U.S. patent application serial number 16 / 354,420, entitled “Dual Mode Controls for Robotic Surgery”;

[0081] U.S. patent application serial number 16 / 354,422, entitled “MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY”;

[0082] U.S. patent application serial number 16 / 354,440, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY”

[0083] U.S. patent application serial number 16 / 354,444, entitled “ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE”

[0084] U.S. patent application serial number 16 / 354,454, entitled “ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS”

[0085] U.S. patent application Ser. No. 16 / 354,461, entitled “SELECTABLE VARIABLE RESPONSEOF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS”

[0086] U.S. patent application serial number 16 / 354,470, entitled “SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS”

[0087] U.S. patent application serial number 16 / 354,474, entitled “ROBOTIC SURGICAL CONTROLSHAVING FEEDBACK CAPABILITIES”

[0088] U.S. patent application Ser. No. 16 / 354,478, entitled “ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK”; and

[0089] U.S. patent application serial number 16 / 354,481, entitled “JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS.”

[0090] The applicant of the present application also owns the following U.S. patent applications filed on September 11, 2018, each of which is incorporated herein by reference in its entirety:

[0091] U.S. patent application serial number 16 / 128,179, entitled “SURGICAL VISUALIZATION PLATFORM”;

[0092] U.S. patent application serial number 16 / 128,180, entitled “CONTROLLING AN EMITTERASSEMBLY PULSE SEQUENCE”;

[0093] U.S. patent application serial number 16 / 128,198, entitled “SINGULAR EMR SOURCE EMITTER ASSEMBLY”;

[0094] U.S. patent application serial number 16 / 128,207, entitled “COMBINATION EMITTER AND CAMERA ASSEMBLY”;

[0095] U.S. patent application serial number 16 / 128,176, entitled “SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES”;

[0096] U.S. patent application serial number 16 / 128,187, entitled “Surgical Visalization of Multiple Targets”;

[0097] U.S. patent application serial number 16 / 128,192, entitled “VISUALIZATION OF SURGICALDEVICES”;

[0098] U.S. patent application serial number 16 / 128,163, entitled “OPERATIVE COMMUNICATION OFLIGHT”;

[0099] U.S. patent application serial number 16 / 128,197, entitled “ROBOTIC LIGHT PROJECTION TOOLS”;

[0100] U.S. patent application serial number 16 / 128,164, entitled “SURGICAL VISUALIZATION FEEDBACK SYSTEM”;

[0101] U.S. patent application serial number 16 / 128,193, entitled “SURGICAL VISUALIZATION AND MOITORING”;

[0102] U.S. patent application serial number 16 / 128,195, entitled “INTEGRATION OF IMAGING DATA”;

[0103] U.S. patent application serial number 16 / 128,170, entitled “ROBOTICALLY-ASSISTED SURGICALSUTURING SYSTEMS”;

[0104] U.S. patent application serial number 16 / 128,183, entitled “SAFETY LOGIC FOR SURGICALSUTURING SYSTEMS”;

[0105] U.S. patent application serial number 16 / 128,172, entitled “ROBOTIC SYSTEM WITH SEPARATEPHOTOACOUSTIC RECEIVER”; and

[0106] U.S. patent application serial number 16 / 128,185, entitled “FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION.”

[0107] The applicant of the present application also owns the following U.S. patent applications filed on March 29, 2018, each of which is incorporated herein by reference in its entirety:

[0108] U.S. patent application serial number 15 / 940,627, entitled “DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS,” now U.S. Patent Application Publication No. 2019 / 0201111;

[0109] U.S. patent application serial number 15 / 940,676, entitled “AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS,” now U.S. Patent Application Publication No. 2019 / 0201142;

[0110] U.S. patent application Ser. No. 15 / 940,711, entitled “SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS,” now U.S. Patent Application Publication No. 2019 / 0201120; and

[0111] U.S. patent application serial number 15 / 940,722, entitled “CHARACTERIZATION OF TISSUEIR REGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY,” now U.S. Patent Application Publication No. 2019 / 0200905.

[0112] The applicant of this patent application owns the following U.S. patent applications filed on December 4, 2018, the disclosures of each of these provisional patent applications are incorporated herein by reference in their entirety:

[0113] U.S. patent application serial number 16 / 209,395, entitled “METHOD OF HUB COMMUNICATION,” now U.S. Patent Application Publication No. 2019 / 0201136;

[0114] U.S. patent application serial number 16 / 209,403, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” now U.S. Patent Application Publication No. 2019 / 0206569;

[0115] U.S. patent application serial number 16 / 209,407, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL,” now U.S. Patent Application Publication No. 2019 / 0201137;

[0116] U.S. patent application serial number 16 / 209,416, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS,” now U.S. Patent Application Publication No. 2019 / 0206562;

[0117] U.S. patent application Ser. No. 16 / 209,423, entitled “METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS,” now U.S. Patent Application Publication No. 2019 / 0200981;

[0118] U.S. patent application Ser. No. 16 / 209,427, entitled “METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIOFREQUENCY DEVICES,” now U.S. Patent Application Publication No. 2019 / 0208641;

[0119] U.S. patent application Ser. No. 16 / 209,433, entitled “METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB,” now U.S. Patent Application Publication No. 2019 / 0201594;

[0120] U.S. patent application serial number 16 / 209,447, entitled “METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB,” now U.S. Patent Application Publication No. 2019 / 0201045;

[0121] U.S. patent application serial number 16 / 209,453, entitled “METHOD FOR CONTROLLING SMARTENERGY DEVICES,” now U.S. Patent Application Publication No. 2019 / 0201046;

[0122] U.S. patent application serial number 16 / 209,458, entitled “METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE,” now U.S. Patent Application Publication No. 2019 / 0201047;

[0123] U.S. patent application Ser. No. 16 / 209,465, entitled “METHOD FOR ADAPTIVE CONTROLSCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION,” now U.S. Patent Application Publication No. 2019 / 0206563;

[0124] U.S. patent application Ser. No. 16 / 209,478, entitled “METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE,” now U.S. Patent Application Publication No. 2019 / 0104919;

[0125] U.S. patent application Ser. No. 16 / 209,490, entitled “METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION,” now U.S. Patent Application Publication No. 2019 / 0206564; and

[0126] U.S. patent application serial number 16 / 209,491, entitled “METHOD FOR CIRCULAR STAPLERCONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS,” now U.S. Patent Application Publication No. 2019 / 0200998.

[0127] Before describing in detail various aspects of the surgical visualization platform, it should be noted that the illustrative examples are not limited in application or use to the details of construction and arrangement of components shown in the drawings and the specification. The illustrative examples may be implemented or incorporated in other aspects, variations, and modifications and may be practiced or carried out in various ways. Furthermore, unless otherwise indicated, the terms and expressions used herein are selected for the convenience of the reader and are not intended to be limiting. Furthermore, it should be understood that one or more of the aspects, expressions of aspects, and / or examples described below may be combined with any one or more of the other aspects, expressions of aspects, and / or examples described below.

[0128] Surgical visualization system

[0129] The present disclosure relates to a surgical visualization platform that utilizes "digital surgery" to obtain additional information about a patient's anatomy and / or surgical procedure. The surgical visualization platform is further configured to communicate the data and / or information to one or more clinicians in a helpful manner. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomy and / or surgical procedure.

[0130] "Digital surgery" can encompass robotic systems, advanced imaging, advanced instrumentation, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and more. Although the various surgical visualization platforms described herein can be used in conjunction with robotic surgical systems, the surgical visualization platforms are not limited to use with robotic surgical systems. In some cases, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0131] In some cases, a surgical system incorporating a surgical visualization platform can implement intelligent dissection to identify and avoid critical structures. Critical structures include anatomical structures such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve and / or tumors. In other cases, the critical structure can be, for example, an alien structure in the dissection field, such as a surgical device, surgical fastener, clamp, tack, bougie, band and / or plate. The critical structure can be determined based on different patients and / or different surgeries. Exemplary critical structures are also described herein. For example, intelligent dissection technology can provide improved intraoperative guidance for dissection and / or key anatomical structure detection and avoidance technology can be used to implement intelligent decision-making.

[0132] Surgical systems incorporating surgical visualization platforms can also implement intelligent anastomosis technology that utilizes improved workflows to provide more consistent anastomosis at optimal locations. Various surgical visualization platforms and procedures described herein can also be utilized to improve cancer localization technology. For example, cancer localization technology can identify and track cancer location, orientation, and its boundaries. In some cases, cancer localization technology can compensate for movement of tools, patients, and / or patient anatomy during surgery to provide guidance to the clinician returning to a point of interest.

[0133] In certain aspects of the present disclosure, a surgical visualization platform can provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue type and health without the need for physical touch, particularly when dissecting and / or placing suturing devices within tissue. Certain tissue characterization techniques described herein can be used without ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, the surgical visualization platform can preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0134] During surgery, the information available to the clinician via the "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures (such as structures embedded or buried within an organ) may be at least partially concealed or hidden from view. Additionally, certain sizes and / or relative distances may be difficult to detect using existing sensor systems and / or difficult to perceive with the "naked eye." Furthermore, certain structures may move preoperatively (e.g., before surgery but after a preoperative scan) and / or intraoperatively. In such cases, the clinician may not be able to accurately determine the location of critical structures intraoperatively.

[0135] When the orientation of a critical structure is uncertain and / or when the proximity between a critical structure and a surgical tool is unknown, the clinician's decision-making process may be hindered. For example, a clinician may avoid certain areas in order to avoid accidentally dissecting a critical structure; however, the avoided area may be unnecessarily large and / or at least partially misplaced. Due to uncertainty and / or excessive / overly cautious operation, the clinician may be unable to enter certain desired areas. For example, excessive caution may cause the clinician to leave behind a portion of a tumor and / or other undesirable tissue in an attempt to avoid a critical structure, even if the critical structure is not in that particular area and / or will not be negatively affected by the clinician working in that particular area. In some cases, surgical results can be improved by increasing knowledge and / or certainty, which can make the surgeon more accurate in specific anatomical areas and, in some cases, make the surgeon less conservative / more aggressive.

[0136] In various aspects, the present disclosure provides surgical visualization systems for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities that go beyond what a clinician can see with the "naked eye" and / or beyond what an imaging system can identify and / or convey to the clinician. Various surgical visualization systems can enhance and strengthen what a clinician can know before tissue treatment (e.g., dissection) and, therefore, can improve outcomes in various situations.

[0137] For example, a visualization system may include a first light emitter configured to emit multiple spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to the spectral waves (spectral imaging), and / or the light pattern. It should be noted that throughout the following disclosure, unless specifically mentioned as visible light, any reference to "light" may include electromagnetic radiation (EMR) or photons in the visible and / or invisible portions of the EMR wavelength spectrum. The surgical visualization system may also include an imaging system and control circuitry in signal communication with the receiver and the imaging system. Based on the output from the receiver, the control circuitry may determine a geometric surface map (i.e., a three-dimensional surface topography) of a visible surface at the surgical site and one or more distances relative to the surgical site. In some cases, the control circuitry may determine one or more distances to at least partially concealed structures. Furthermore, the imaging system may communicate the geometric surface map and the one or more distances to the clinician. In such cases, the enhanced view of the surgical site provided to the clinician may provide a representation of concealed structures within the surroundings of the surgical site. For example, the imaging system can virtually enhance the hidden structure on a geometric surface map of the hidden and / or obstructing tissue, similar to a line drawn on the ground to indicate a practical line below the surface. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to visible obstructing tissue and / or to at least partially obstructed structures and / or the depth of the hidden structure below the visible surface of the obstructing tissue. For example, the visualization system can determine the distance of the enhancement line relative to the surface of the visible tissue and communicate the distance to the imaging system.

[0138] In various aspects of the present disclosure, a surgical visualization system for intraoperative identification and avoidance of critical structures is disclosed. Such a surgical visualization system can provide valuable information to a clinician during a surgical procedure. Thus, for example, the clinician knows that the surgical visualization system is tracking critical structures that can be approached during dissection (such as ureters, specific nerves and / or critical blood vessels), and can confidently maintain momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide instructions to the clinician for a sufficient period of time to enable the clinician to pause and / or slow down the surgical procedure and assess the proximity to the critical structure to prevent accidental damage thereto. The surgical visualization system can provide the clinician with an ideal, optimized and / or customizable amount of information to allow the clinician to confidently and / or quickly move through tissue while avoiding accidental damage to healthy tissue and / or critical structures, and thereby minimizing the risk of injury caused by the surgical procedure.

[0139] Figure 1is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of a critical structure 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In some cases, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near-real-time information regarding proximity data, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured to identify critical structures intraoperatively and / or facilitate surgical device avoidance of critical structures 101. For example, by identifying critical structures 101, a clinician can avoid maneuvering a surgical device around a critical structure 101 and / or an area within a predetermined proximity of a critical structure 101 during a surgical procedure. For example, a clinician can avoid dissecting veins, arteries, nerves, and / or blood vessels identified as critical structures 101 and / or avoid dissecting near such critical structures. In various cases, the critical structures 101 can be determined on a patient-by-patient and / or procedure-by-procedure basis.

[0140] The surgical visualization system 100 incorporates tissue recognition and geometric surface mapping in conjunction with the distance sensor system 104. Combined, these features of the surgical visualization system 100 can determine the position of critical structures 101 within the anatomical field and / or the proximity of the surgical device 102 to the surface 105 of visible tissue and / or to the critical structures 101. Furthermore, the surgical visualization system 100 includes an imaging system comprising, for example, an imaging device 120, such as a camera, configured to provide a real-time view of the surgical site. In various embodiments, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. Views from the imaging device 120 can be provided to the clinician, and in various aspects of the present disclosure, these views can be enhanced with additional information based on tissue recognition, topographic mapping, and the distance sensor system 104. In such cases, the surgical visualization system 100 includes multiple subsystems, namely an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem, which can cooperate to provide clinicians with advanced data synthesis and integrated information during surgery.

[0141] The imaging device may include a camera or imaging sensor configured to detect, for example, visible light, spectral light waves (visible or invisible light), and structured light patterns (visible or invisible light). In various aspects of the present disclosure, the imaging system may include, for example, an imaging device, such as an endoscope. Additionally or alternatively, the imaging system may include, for example, an imaging device, such as an arthroscope, an angioscope, a bronchoscope, a choledochoscope, a colonoscope, a cystoscope, a duodenoscope, an enteroscope, an esophagogastroduodenoscope (gastroscope), a laryngoscope, a nasopharyngeal-nephroscope, a sigmoidoscope, a thoracoscope, a ureteroscope, or an exoscope. In other cases, such as in open surgical applications, the imaging system may not include a viewing scope.

[0142] In various aspects of the present disclosure, the tissue identification subsystem may be implemented using a spectral imaging system. The spectral imaging system may rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled “SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING,” issued on March 1, 2016, which is incorporated herein by reference in its entirety.

[0143] In various aspects of the present disclosure, the surface mapping subsystem can be implemented using a light patterning system, as further described herein. The use of light patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be used in the surgical visualization system described herein.

[0144] Structured light is the process of projecting a known pattern (usually a grid or horizontal stripes) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published on September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, entitled “SET COMPRISING A SURGICAL INSTRUMENT,” published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled “DEPICTION SYSTEM,” published on September 7, 2017, are incorporated herein by reference in their entirety.

[0145] In various aspects of the present disclosure, a distance determination system can be incorporated into a surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of a visible surface and to determine various distances relative to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to tissue (or other structures) identified at a surgical site.

[0146] Figure 2 is a schematic diagram of a control system 133 that may be used with the surgical visualization system 100. The control system 133 includes a control circuit 132 in signal communication with a memory 134. The memory 134 stores instructions executable by the control circuit 132 to determine and / or identify critical structures (e.g., Figure 1 101 ), determine and / or calculate one or more distances and / or three-dimensional digital representations, and transmit certain information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (e.g., Figure 1 The imaging device 120 in FIG. 1 ), one or more displays 146, or one or more controls 148, or any combination thereof. The camera 144 may include one or more image sensors 135 to receive signals from various light sources (e.g., visible light, spectral imagers, 3D lenses, etc.) that emit light in various visible and non-visible light spectrums. The display 146 may include one or more screens or monitors for depicting real, virtual, and / or virtual-augmented images and / or information to one or more clinicians.

[0147] In various aspects, the heart of camera 144 is the image sensor 135. Generally speaking, modern image sensors 135 are solid-state electronic devices containing up to millions of discrete photodetector sites (called pixels). Image sensor 135 technology falls into one of two categories: charge-coupled device (CCD) and complementary metal oxide semiconductor (CMOS) imagers, with short-wave infrared (SWIR) being an emerging imaging technology. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCMOS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 are sensitive to wavelengths from approximately 350 nm to 1050 nm, but this range is often given as 400 nm to 1000 nm. Generally speaking, CMOS sensors are more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, therefore, cannot distinguish colors. Consequently, there are two types of color CCD cameras: single-chip and three-chip. Single-chip color CCD cameras offer a common, low-cost imaging solution and use a mosaic (e.g., Bayer) optical filter to separate the incoming light into a series of colors and employ an interpolation algorithm to resolve the full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras offer higher resolution by employing a prism to direct each portion of the incoming light spectrum to a different chip. More accurate color reproduction is possible because each point in the object's space has a separate RGB intensity value, rather than using an algorithm to determine color. Three-chip cameras offer extremely high resolution.

[0148] The control system 133 also includes a spectral light source 150 and a structured light source 152. In some cases, a single source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, the single light source can be pulsed to provide light in the invisible spectrum (e.g., infrared spectrum light) and wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various cases, the tissue identification logic component 140 can identify key structures via data from the spectral light source 150 received in part by the image sensor 135 of the camera 144. The surface mapping logic component 136 can determine the surface contour of the visible tissue based on the reflected structured light. Using the time of flight measurement results, the distance determination logic component 141 can determine one or more distances to the visible tissue and / or key structure 101. One or more outputs from the surface mapping logic 136 , tissue identification logic 140 , and distance determination logic 141 may be provided to the imaging logic 138 and may be combined, blended, and / or overlaid for communication to the clinician via a display 146 of the imaging system 142 .

[0149] The manual now briefly turns to Figures 2A to 2C , to describe various aspects of the control circuitry 132 for controlling various aspects of the surgical visualization system 100. Figure 2A , shows a control circuit 400 configured to control various aspects of the surgical visualization system 100 according to at least one aspect of the present disclosure. The control circuit 400 can be configured to implement the various processes described herein. The control circuit 400 can include a microcontroller comprising one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions to implement the various processes described herein. The processor 402 can be any of a variety of single-core or multi-core processors known in the art. The memory circuit 404 can include volatile storage media and non-volatile storage media. The processor 402 can include an instruction processing unit 406 and an operation unit 408. The instruction processing unit can be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0150] Figure 2B 1. Combinatorial logic circuit 410 is shown that is configured to control various aspects of surgical visualization system 100 in accordance with at least one aspect of the present disclosure. Combinatorial logic circuit 410 can be configured to implement various processes described herein. Combinatorial logic circuit 410 can include a finite state machine that includes a combinatorial logic component 412 that is configured to receive data associated with a surgical instrument or tool at an input 414, process the data via combinatorial logic component 412, and provide an output 416.

[0151] Figure 2C A sequential logic circuit 420 is shown that is configured to control various aspects of the surgical visualization system 100 in accordance with at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic component 422 may be configured to implement the various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, a combinational logic component 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 may store a current state of the finite state machine. In some cases, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic component 422 is configured to receive data associated with a surgical device or system from an input 426, process the data through the combinational logic component 422, and provide an output 428. In other aspects, the circuit may include a processor (e.g., Figure 2AIn other aspects, the finite state machine may include a combinational logic circuit (e.g., combinational logic circuit 410, Figure 2B ) and a combination of a sequential logic circuit 420.

[0152] See again Figure 1 In the surgical visualization system 100 of the present invention, the key structure 101 can be an anatomical structure of interest. For example, the key structure 101 can be an anatomical structure such as a ureter, an artery such as the superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor. In other cases, the key structure 101 can be, for example, a foreign structure in the anatomical field, such as a surgical device, a surgical fastener, a clamp, a tack, a bougie, a band, and / or a plate. Exemplary key structures are further described herein and in the aforementioned concurrently filed U.S. patent applications (including, for example, U.S. patent application Ser. No. 16 / 128,192, filed on September 11, 2018, entitled “VISUALIZATION OF SURGICAL DEVICES”), which are incorporated herein by reference in their entirety.

[0153] In one aspect, the critical structure 101 can be embedded in the tissue 103. In other words, the critical structure 101 can be positioned below the surface 105 of the tissue 103. In such cases, the tissue 103 conceals the critical structure 101 from the clinician's view. From the perspective of the imaging device 120, the critical structure 101 is also obscured by the tissue 103. The tissue 103 can be, for example, fat, connective tissue, adhesions, and / or an organ. In other cases, the critical structure 101 can be partially obscured from view.

[0154] Figure 1 Also depicted is a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from a distal end of a shaft of the surgical device 102. The surgical device 102 can be any suitable surgical device, such as, for example, a dissector, a stapler, a grasper, a clip applier, and / or an energy device (including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector). Additionally or alternatively, the surgical device 102 can include, for example, another imaging or diagnostic modality, such as an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 can be configured to enable identification of one or more critical structures 101 and the proximity of the surgical device 102 to the critical structure 101.

[0155] The imaging device 120 of the surgical visualization system 100 is configured to be capable of detecting various wavelengths of light, such as, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 may also include a waveform sensor 122 (such as a spectral image sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens that are used together to simultaneously record two two-dimensional images, and thereby generate a three-dimensional image of the surgical site, render a three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to be capable of receiving images indicating the topography of visible tissue and the identification and orientation of hidden critical structures, as further described herein. For example, the field of view of the imaging device 120 may overlap with the pattern of light (structured light) on the surface 105 of the tissue, such as Figure 1 shown.

[0156] In one aspect, the surgical visualization system 100 can be incorporated into a robotic system 110. For example, the robotic system 110 can include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include rigid structural members 116 and joints 118, which can include servo motor controls. The first robotic arm 112 is configured to manipulate the surgical device 102, and the second robotic arm 114 is configured to manipulate the imaging device 120. A robotic control unit can be configured to issue control motions to the robotic arms 112, 114, which can affect, for example, the surgical device 102 and the imaging device 120.

[0157] The surgical visualization system 100 also includes an emitter 106 configured to emit a pattern of light, such as stripes, grid lines, and / or dots, to enable determination of the topography or terrain of the surface 105. For example, a projected light array 130 can be used for three-dimensional scanning and registration of the surface 105. The projected light array 130 can be emitted from the emitter 106 located, for example, on the surgical device 102 and / or one of the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is used to determine the shape defined by the surface 105 of the tissue 103 and / or the intraoperative motion of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances relative to the surface 105.

[0158] In one aspect, the imaging device 120 may further include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effects of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable identification of the type of anatomical and / or physical structure (such as the critical structure 101). Identification of the critical structure 101 may be achieved, for example, through spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 may comprise, for example, a multispectral imaging system and / or a selective spectral imaging system. In other cases, the waveform sensor 122 and the optical waveform emitter 123 may comprise, for example, a photoacoustic imaging system. In other cases, the optical waveform emitter 123 may be located on a surgical device separate from the imaging device 120 .

[0159] The surgical visualization system 100 may also include a distance sensor system 104 that is configured to determine one or more distances at the surgical site. In one aspect, the time of flight distance sensor system 104 can be a time of flight distance sensor system that includes an emitter, such as emitter 106, and a receiver 108 that can be positioned on the surgical device 102. In other cases, the time of flight emitter can be separate from the structured light emitter. In one general aspect, the emitter 106 portion of the time of flight distance sensor system 104 can include a very tiny laser source, and the receiver 108 portion of the time of flight distance sensor system 104 can include a matching sensor. The time of flight distance sensor system 104 can detect the "time of flight" or the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. The use of a very narrow light source in the emitter 106 enables the distance sensor system 104 to determine the distance to the surface 105 of the tissue 103 directly in front of the distance sensor system 104. Still referring to Figure 1 , d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue. The distance sensor system 104 can be used to determine the transmitter-tissue distance d e Device-tissue distance d tThe device-tissue distance d can be obtained based on the known position of the transmitter 106 on the axis of the surgical device 102 relative to the distal end of the surgical device 102. In other words, when the distance between the transmitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t The transmitter-tissue distance d e In some cases, the shaft of surgical device 102 may include one or more articulation joints and may be capable of articulation relative to transmitter 106 and jaws. The articulation configuration may include, for example, a multi-jointed vertebra-like structure. In some cases, a three-dimensional camera may be used to triangulate one or more distances to surface 105.

[0160] In various instances, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate surgical device rather than on the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar through which the surgical device 102 extends to reach the surgical site. In other instances, the receiver 108 of the time-of-flight distance sensor system 104 may be mounted on a separate robotically controlled arm (e.g., robotic arm 114), on a movable arm operated by another robot, and / or mounted to an operating room (OR) table or fixture. In some instances, the imaging device 120 includes the time-of-flight receiver 108 to determine the distance from the transmitter 106 on the surgical device 102 to the surface 105 of the tissue 103 using a line between the transmitter 106 on the surgical device 102 and the imaging device 120. For example, the distance d may be determined based on the known positions of the transmitter 106 (on the surgical device 102) and the receiver 108 (on the imaging device 120) of the time-of-flight distance sensor system 104. e The three-dimensional position of the receiver 108 may be known and / or registered intraoperatively with the robot coordinate plane.

[0161] In some cases, the position of the transmitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other cases, the surgical visualization system 100 can be used separately from the robotic system. In such cases, the distance sensor system 104 can be independent of the robotic system.

[0162] In some cases, one or more of the robotic arms 112, 114 may be separate from the main robotic system used during the surgical procedure. At least one of the robotic arms 112, 114 may be positioned and registered to a specific coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 may control and / or register the position of the robotic arms 112, 114 relative to a specific coordinate system. Similarly, the position of the surgical device 102 and the imaging device 120 may be registered relative to a specific coordinate system.

[0163] Still see Figure 1 , d w is the camera-to-key structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the key structure 101, and d A is the depth of the critical structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the critical structure 101). In various aspects, the time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-critical structure distance d w The use of spectral imaging in conjunction with a time-of-flight sensor is further described herein. Furthermore, see now Figure 3 In various aspects of the present disclosure, the depth d of the critical structure 101 relative to the surface 105 of the tissue 103 is A It can be determined by the following method: According to the distance d w and the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and therefore the known distance d between them). x ) to perform triangulation to determine the distance d y (It is the distance d e and d A sum).

[0164] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, the first waveform (or range of waveforms) can be used to determine the camera-critical structure distance d w , and the second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such cases, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0165] Additionally or alternatively, in some cases, the distance d A It can be determined by ultrasound, registered magnetic resonance imaging (MRI), or computed tomography (CT) scans. In other cases, the distance dA This can be determined using spectral imaging because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first manner or amount, and collagen can reduce the detection signal in a different second manner or amount.

[0166] Now see Figure 4 The surgical visualization system 160 in FIG. 1 includes a surgical device 162 including an optical waveform emitter 123 and a waveform sensor 122 configured to detect a reflected waveform. The optical waveform emitter 123 can be configured to emit a waveform for determining a distance d from a common device such as the surgical device 162. t and d w , as further described herein. In such cases, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 is A It can be determined as follows:

[0167] d A =d w -d t .

[0168] As disclosed herein, various information about visible tissue, embedded critical structures, and surgical devices can be determined by utilizing a combined approach that combines an image sensor configured to detect spectral wavelengths and structured light arrays with one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays. Furthermore, the image sensor can be configured to receive visible light and, thus, provide an image of the surgical site to the imaging system. Logic or algorithms are employed to identify the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light, and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various embodiments, the imaging device 120 may include multiple image sensors.

[0169] Camera-key structure distance d w Detection can also be performed in one or more alternative ways. In one aspect, the key structure 201 can be illuminated using, for example, fluoroscopic visualization techniques such as fluorescent indocyanine green (ICG), such as Figures 6 to 8 The camera 220 may include two optical waveform sensors 222 and 224, which simultaneously capture the left image and the right image of the key structure 201 ( Figure 7A and Figure 7B ). In such cases, the camera 220 may depict the glow of the critical structure 201 below the surface 205 of the tissue 203 and the distance d wThe distance between sensors 222 and 224 can be determined based on the known distance between them. In some cases, the distance can be more accurately determined by utilizing more than one camera or by moving the camera between multiple locations. In some aspects, one camera can be controlled by a first robotic arm, and the second camera can be controlled by another robotic arm. In such a robotic system, one camera can be, for example, a slave camera on a slave arm. The slave arm and its camera can be programmed to track the other camera and maintain, for example, a specific distance and / or lens angle.

[0170] In other aspects, the surgical visualization system 100 may employ two separate waveform receivers (ie, cameras / image sensors) to determine d w Now see Figure 9 If a critical structure 301 or its contents (eg, a vessel or vessel contents) can emit a signal 302, such as using fluoroscopy, the actual location can be triangulated from two separate cameras 320a, 320b at known locations.

[0171] On the other hand, see now Figure 10A and Figure 10B The surgical visualization system may use shaking or moving the camera 440 to determine the distance d w The camera 440 is robotically controlled so that the three-dimensional coordinates of the camera 440 at different positions are known. In various cases, the camera 440 can be pivoted at the cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., a blood vessel or blood vessel contents) can emit a signal, such as using fluoroscopy, the actual position can be triangulated based on the camera 440 being rapidly moved between two or more known positions. Figure 10A , camera 440 is moved axially along axis A. More specifically, camera 440 is translated a distance d1 along axis A closer to key structure 401 to a position indicated as position 440', such as by moving in and out on a robotic arm. As camera 440 moves distance d1 and the size of the view changes relative to key structure 401, the distance to key structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d1) can correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, camera 440 can be rotated or swept along an arc between different orientations. Referring now to Figure 10B , the camera 440 moves axially along axis A and rotates around axis A by an angle θ3. The pivot point 442 for the rotation of the camera 440 is located at the cannula / patient interface. Figure 10B In FIG. 4 , the camera 440 is translated and rotated to position 440 . When the camera 440 is moved and the edge of the view changes with respect to the key structure 401 , the distance to the key structure 401 can be calculated. Figure 10B, the distance d2 may be, for example, 9.01 mm, and the angle θ3 may be, for example, 0.9 degrees.

[0172] Figure 5 A surgical visualization system 500 is depicted, which is similar in many respects to surgical visualization system 100. In various embodiments, surgical visualization system 500 can be another example of surgical visualization system 100. Similar to surgical visualization system 100, surgical visualization system 500 includes a surgical device 502 and an imaging device 520. Imaging device 520 includes a spectral light emitter 523 configured to emit spectral light at multiple wavelengths to obtain, for example, spectral images of hidden structures. In various embodiments, imaging device 520 can also include a three-dimensional camera and associated electronic processing circuitry. Surgical visualization system 500 is shown being used intraoperatively to identify and facilitate avoidance of certain critical structures that are not visible on the surface, such as ureters 501a and blood vessels 501b in organ 503 (in this example, the uterus).

[0173] The surgical visualization system 500 is configured to be able to determine an emitter-tissue distance d from an emitter 506 on a surgical device 502 to a surface 505 of a uterus 503 via structured light. e The surgical visualization system 500 is configured to be able to detect the presence of a transducer based on the transmitter-tissue distance d e To extrapolate the device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 t The surgical visualization system 500 is further configured to determine the tissue-ureter distance d from the ureter 501a to the surface 505. A and the camera-ureter distance d from the imaging device 520 to the ureter 501a w As this article about Figure 1 As described above, for example, the surgical visualization system 500 can utilize, for example, spectral imaging and a time-of-flight sensor to determine the distance d w In various circumstances, the surgical visualization system 500 can determine (eg, triangulate) the tissue-ureter distance d based on other distance and / or surface mapping logic described herein. A (or depth).

[0174] Now see Figure 11, which depicts a schematic diagram of a control system 600 for, for example, a surgical visualization system (such as surgical visualization system 100). For example, control system 600 is a conversion system that integrates spectral signature tissue recognition and structured light tissue localization to identify key structures, particularly when these structures are obscured by other tissues (such as fat, connective tissue, blood, and / or other organs). Such techniques can also be used to detect tissue variability, such as distinguishing tumors and / or unhealthy tissue from healthy tissue within an organ.

[0175] The control system 600 is configured to implement a hyperspectral imaging and visualization system that utilizes molecular responses to detect and identify anatomical structures within the surgical field of view. The control system 600 includes conversion logic 648 to convert tissue data into information usable by the surgeon. For example, variable reflectivity based on wavelength relative to an obscuring material can be used to identify critical structures within the anatomy. Furthermore, the control system 600 combines the identified spectral signatures with structured light data within an image. For example, the control system 600 can be used to create a three-dimensional dataset for surgical applications in a system with enhanced image overlays. The technology can be used both intraoperatively and preoperatively with the additional visual information. In various scenarios, the control system 600 is configured to provide a warning to the clinician when approaching one or more critical structures. Various algorithms can be employed to guide robotic automation and semi-automated approaches based on the surgical procedure and proximity to critical structures.

[0176] Projected light arrays are used to determine tissue shape and motion intraoperatively. Alternatively, flash lidar can be used for surface mapping of tissue.

[0177] The control system 600 is configured to detect critical structures and provide image overlays of the critical structures, and to measure distances to the surface of visible tissue and distances to embedded / buried critical structures. In other cases, the control system 600 may measure distances to the surface of visible tissue or detect critical structures and provide image overlays of the critical structures.

[0178] The control system 600 includes a spectrum control circuit 602. For example, the spectrum control circuit 602 may be a field programmable gate array (FPGA) or a Figures 2A to 2CAnother suitable circuit configuration is described. Spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. For example, processor 604 can be configured for hyperspectral processing and can utilize C / C++ code. For example, video input processor 606 receives video input including control (metadata) data, such as shutter time, wavelength, and sensor analysis. Processor 604 is configured to process the video input signal from video input processor 606 and provide a video output signal to video output processor 608, which includes, for example, a hyperspectral video output that interfaces with control (metadata) data. Video output processor 608 provides the video output signal to image overlay controller 610.

[0179] The video input processor 606 is coupled to a camera 612 at the patient side via a patient isolation circuit 614. As previously described, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit may include multiple transformers to isolate the patient from other circuits in the system. The camera 612 receives intraoperative images through optics 632 and an image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or may include, for example, a CMOS image sensor. Figure 2 Any image sensor technology described herein. In one aspect, camera 612 outputs images at a 14-bit / pixel signal. It should be understood that higher or lower pixel resolutions may be employed without departing from the scope of this disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which processes the camera output signal 613 using hardware registers 618 and a Nios2 coprocessor 620. The color RGB fused output signal is provided to a video input processor 606 and a laser pulse control circuit 622.

[0180] The laser pulse control circuit 622 controls the laser engine 624. The laser engine 624 outputs multiple wavelengths (λ1, λ2, λ3, λ4, λ5, λ6, λ7, λ8, λ9, λ10, λ11, λ22, λ33, λ12, λ13, λ14, λ15, λ16, λ17, λ18, λ19, n ) light. Laser engine 624 can operate in multiple modes. In one aspect, laser engine 624 can operate in two modes, for example. In a first mode (e.g., normal operating mode), laser engine 624 outputs an illumination signal. In a second mode (e.g., recognition mode), laser engine 624 outputs RGBG and NIR light. In various cases, laser engine 624 can operate in a polarization mode.

[0181] Light output 626 from the laser engine 624 illuminates the target anatomical structure in the intraoperative surgical site 627. The laser pulse control circuit 622 also controls a laser pulse controller 628 for a laser pattern projector 630, which projects a laser pattern 631 (such as a grid or pattern of lines and / or dots) of a predetermined wavelength (λ2) onto the surgical tissue or organ at the surgical site 627. The camera 612 receives the patterned light and the reflected light output by the camera optics 632. The image sensor 634 converts the received light into a digital signal.

[0182] The color RGB fusion circuit 616 also outputs a signal to the image overlay controller 610 and the video input module 636 for reading the laser pattern 631 projected by the laser pattern projector 630 onto the target anatomical structure at the surgical site 627. The processing module 638 processes the laser pattern 631 and outputs a first video output signal 640 representing the distance to the visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0183] The first video output signal 640 and the second video output signal 642 include data representing the location of the critical structure on the three-dimensional surface model, which is provided to the integration module 643. In combination with the data from the video output processor 608 of the spectral control circuit 602, the integration module 643 can determine the distance d to the buried critical structure. A ( Figure 1 ) (e.g., via triangulation algorithm 644), and the distance d A The image overlay controller 610 may be provided via the video output processor 646. The conversion logic components described above may include the conversion logic circuit 648, the intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0184] In various cases, preoperative data 650 from a CT or MRI scan may be used to register or match certain three-dimensional deformable tissues. Such preoperative data 650 may be provided to an integration module 643 and ultimately to an image overlay controller 610 so that such information may be overlaid with the view from a camera 612 and provided to a video monitor 652. Registration of preoperative data is further described herein and in the aforementioned concurrently filed U.S. patent applications (including, for example, U.S. patent application Ser. No. 16 / 128,195, filed Sep. 11, 2018, entitled "INTEGRATION OF IMAGING DATA"), which are incorporated herein by reference in their entirety.

[0185] The video monitor 652 can output the integrated / enhanced view from the image overlay controller 610. The clinician can select and / or switch between different views on one or more monitors. On the first monitor 652a, the clinician can switch between (A) a view in which a three-dimensional rendering of visible tissue is depicted and (B) an enhanced view in which one or more hidden critical structures are depicted on the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can, for example, switch distance measurements to the surface of one or more hidden critical structures and / or visible tissue.

[0186] The control system 600 and / or its various control circuits may be incorporated into the various surgical visualization systems disclosed herein.

[0187] Figure 12 7. A structured (or patterned) light system 700 is shown in accordance with at least one aspect of the present disclosure. As described herein, structured light in the form of stripes or lines, for example, can be projected from a light source and / or projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contours of the surface 705. Figure 1 ) can be configured to detect a projected pattern of light on surface 705. The way in which the projected pattern deforms when it strikes surface 705 allows the vision system to calculate depth and surface information of the target anatomy.

[0188] In some cases, invisible (or imperceptible) structured light can be used without interfering with other computer vision tasks where the projected pattern might confuse the image. For example, infrared light that alternates between two diametrically opposed patterns or extremely fast visible light frame rates can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0189] As described above, the various surgical visualization systems described herein can be used to visualize various different types of tissue and / or anatomical structures, including tissue and / or anatomical structures that may be obscured from visualization by EMR in the visible portion of the spectrum. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissue based on different combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the tissue's absorption coefficient at various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the visualized tissue based on a particular combination of constituent materials. To illustrate, Figure 13A23 is a graph 2300 depicting how the absorption coefficient of various biological materials varies across the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 represents the absorption coefficient of the biological material (e.g., in cm -1 ), and horizontal axis 2304 represents EMR wavelength (e.g., in μm). Graph 2300 further illustrates a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxyhemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxyhemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials, and thus the tissue types visualized by the surgical visualization system can be identified and distinguished based on the specific combination of constituent materials detected. Accordingly, the spectral imaging system can be configured to emit EMR at multiple different wavelengths, determine the constituent materials of the tissue based on the absorbed EMR responses detected at the different wavelengths, and then characterize the tissue type based on the specific detected combination of constituent materials.

[0190] Figure 13B The use of spectral imaging techniques to visualize different tissue types and / or anatomical structures is shown. Figure 13B In FIG. 2 , a spectral emitter 2320 (e.g., spectral light source 150) is used by the imaging system to visualize a surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from tissue and / or structures at the surgical site 2325 can be imaged by the image sensor 135 ( Figure 2 ) to visualize tissue and / or structures; the tissue and / or structure may be visible (e.g., located on the surface of the surgical site 2325) or obscured (e.g., located beneath other tissue and / or structures at the surgical site 2325). In this example, the imaging system 142 ( Figure 2 ) can visualize tumors 2332, arteries 2334, and various abnormalities 2338 (i.e., tissue that does not conform to a known or expected spectral signature) based on spectral signatures characterized by the different absorption properties (e.g., absorption coefficients) of the constituent materials of each of the different tissue / structure types. The visualized tissues and structures can be displayed on a display associated with or coupled to the imaging system 142, such as the imaging system display 146 ( Figure 2 )、Main display 2119( Figure 18 ), non-sterile display 2109 ( Figure 18 ), hub display 2215( Figure 19)、Device / Equipment Display 2237( Figure 19 )wait.

[0191] Furthermore, the imaging system 142 can be configured to customize or update the displayed visualization of the surgical site based on the identified tissue and / or structure type. For example, the imaging system 142 can display a margin 2330a associated with the tumor 2332 being visualized on a display screen (e.g., display 146). The margin 2330a can indicate the area or amount of tissue that should be removed to ensure complete removal of the tumor 2332. The control system 133( Figure 2 ) can be configured to control or update the size of edge 2330a based on tissue and / or structure identified by imaging system 142. In the illustrated example, imaging system 142 has identified multiple anomalies 2338 within the FOV. Accordingly, control system 133 can adjust displayed edge 2330a to a first updated edge 2330b, which is of sufficient size to encompass anomalies 2338. Additionally, imaging system 142 has identified an artery 2334 that partially overlaps with initially displayed edge 2330a (as indicated by highlighted region 2336 of artery 2334). Accordingly, control system 133 can adjust displayed edge 2330a to a second updated edge 2330c, which is of sufficient size to encompass the relevant portion of artery 2334.

[0192] In addition to the above Figure 13A and 13B In addition to or instead of the absorption properties described, tissues and / or structures can also be imaged or characterized based on their reflectance properties at the EMR wavelength spectrum. For example, Figures 13C-13E Various graphs showing the reflectivity of different types of tissues or structures at different EMR wavelengths. Figure 13C is a graphical representation 1050 of an exemplary ureteral feature relative to a mask. Figure 13D is a graphical representation 1052 of an exemplary artery feature relative to an obscuration. Figure 13E is a graphical representation 1054 of an exemplary neural feature relative to an obstruction. Figures 13C-13E The curves in FIG represent the reflectance of specific structures (ureters, arteries, and nerves) relative to the corresponding reflectance of fat, lung tissue, and blood at corresponding wavelengths as a function of wavelength (nm). These graphs are for illustrative purposes only, and it should be understood that other tissues and / or structures may have corresponding detectable reflectance characteristics that will allow identification and visualization of the tissues and / or structures.

[0193] In various cases, selected wavelengths for spectral imaging (i.e., "selective spectral" imaging) can be identified and utilized based on anticipated critical structures and / or obstructions at the surgical site. By utilizing selective spectral imaging, the amount of time required to obtain spectral images can be minimized, allowing information to be obtained in real time or near real time and utilized during surgery. In various cases, the wavelength can be selected by the clinician or by the control circuit based on the clinician's input. In some cases, the wavelength can be selected based on, for example, machine learning and / or big data that the control circuit can access via the cloud.

[0194] The aforementioned application of spectral imaging to tissue can be used intraoperatively to measure the distance between a waveform emitter and a critical structure obscured by tissue. In one aspect of the present disclosure, referring now to Figure 14 and Figure 15 , showing a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125. In some cases, the time-of-flight sensor system 1104 may be incorporated into the surgical visualization system 100 ( Figure 1 ). Time-of-flight sensor system 1104 includes a waveform transmitter 1106 and a waveform receiver 1108 on the same surgical device 1102. Transmitted wave 1124 extends from transmitter 1106 to critical structure 1101, and received wave 1125 is reflected back from critical structure 1101 by receiver 1108. Surgical device 1102 is positioned through a trocar 1110 that extends into a cavity 1107 of a patient.

[0195] The waveforms 1124, 1125 are configured to be able to penetrate the obscured tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 may be in the NIR or SWIR wavelength spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectral) or a photoacoustic signal may be emitted from the emitter 1106 and may penetrate the tissue 1103 in which the critical structure 1101 is concealed. The emitted waveform 1124 may be reflected by the critical structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various cases, the waveforms 1124, 1125 may be selected based on the spectral characteristics of the critical structure 1101 to target the critical structure 1101 within the tissue 1103, as further described herein. In various cases, the emitter 1106 is configured to be able to provide a binary signal on and off, such as Figure 15 As shown, for example, the binary signal may be measured by receiver 1108 .

[0196] Based on the delay between the transmitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to be able to determine the distance d ( Figure 14 ). Figure 14The time-of-flight timing diagram 1130 of the transmitter 1106 and receiver 1108 is shown in Figure 15 The delay is a function of the distance d, and the distance d is given by:

[0197]

[0198] in:

[0199] c = speed of light;

[0200] t = length of the pulse;

[0201] q1 = charge accumulated when light is emitted; and

[0202] q2 = Charge accumulated when no light is emitted.

[0203] As provided herein, the time of flight of waveforms 1124, 1125 corresponds to Figure 14 In various cases, additional transmitter / receivers and / or pulsed signals from transmitter 1106 can be configured to transmit non-penetrating signals. Non-tissue penetrating signals can be configured to determine the distance from the transmitter to the surface 1105 of the obscuring tissue 1103. In various cases, the depth of the critical structure 1101 can be determined by the following formula:

[0204] d A =d w -d t .

[0205] in:

[0206] d A = depth of key structure 1101;

[0207] d w = distance from transmitter 1106 to key structure 1101 ( Figure 14 d) in the above clause; and

[0208] d t = the distance from the emitter 1106 (on the distal end of the surgical device 1102 ) to the surface 1105 of the shielding tissue 1103 .

[0209] In one aspect of the present disclosure, see now Figure 16 , shows a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c. In some cases, the time-of-flight sensor system 1204 can be incorporated into the surgical visualization system 100 ( Figure 1). The time-of-flight sensor system 1204 includes a waveform transmitter 1206 and a waveform receiver 1208. The waveform transmitter 1206 is positioned on the first surgical device 1202a, and the waveform receiver 1208 is positioned on the second surgical device 1202b. The surgical devices 1202a and 1202b are positioned through their respective trocars 1210a and 1210b, respectively, which extend into a cavity 1207 of a patient. Transmitted waves 1224a, 1224b, and 1224c extend from the transmitter 1206 toward the surgical site, and received waves 1225a, 1225b, and 1225c are reflected from various structures and / or surfaces at the surgical site back to the receiver 1208.

[0210] Different emission waves 1224a, 1224b, 1224c are configured to target different types of materials at the surgical site. For example, wave 1224a targets shielding tissue 1203, wave 1224b targets a first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets a second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, 1224c can be in the visible light, NIR, or SWIR wavelength spectrum. For example, visible light can be reflected from the surface 1205 of the tissue 1203, and NIR and / or SWIR waveforms can be configured to penetrate the surface 1205 of the tissue 1203. In various aspects, as described herein, a spectral signal (e.g., a hyperspectral, multispectral, or selective spectral) or a photoacoustic signal can be emitted from the emitter 1206. In various cases, waves 1224b, 1224c may be selected to target critical structures 1201a, 1201b within tissue 1203 based on spectral characteristics of the critical structures 1201a, 1201b, as further described herein. Photoacoustic imaging is further described in various US patent applications, which are incorporated by reference into this disclosure.

[0211] The transmitted waves 1224a, 1224b, 1224c may be reflected from the target material (ie, the surface 1205, the first critical structure 1201a, and the second critical structure 1201b, respectively). The received waveforms 1225a, 1225b, 1225c may be due to Figure 16 The distance d shown 1a d 2a d 3a d 1b d 2b d 2c and was delayed.

[0212] In a time-of-flight sensor system 1204 in which the transmitter 1206 and receiver 1208 are independently positionable (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d may be calculated based on the known positions of the transmitter 1206 and receiver 1208. 1a d 2a d 3a d 1b d 2b d 2c For example, when the surgical devices 1202a, 1202b are robotically controlled, these positions may be known. Knowledge of the positions of the emitter 1206 and receiver 1208, as well as the timing of the photon stream targeting a tissue and the specific response received by the receiver 1208, may allow the distance d to be determined. 1a d 2a d 3a d 1b d 2b d 2c In one aspect, the distance to the obscured critical structures 1201a, 1201b can be triangulated using the penetration wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, the time-of-flight sensor system 1204 can determine various distances.

[0213] Still see Figure 16 In various cases, the receiver 1208 can be rotated so that the center of mass of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b, in the view provided to the clinician. Such an orientation can quickly convey one or more relevant distances and / or viewing angles with respect to the critical structure. For example, Figure 16 As shown, the surgical site is displayed from a perspective of key structures 1201a perpendicular to the viewing plane (i.e., with the vessels oriented in-page / out-of-page). In various cases, such an orientation may be the default setting; however, the view may be rotated or otherwise adjusted by the clinician. In some cases, the clinician may switch between different surfaces and / or target structures that define the perspective of the surgical site provided by the imaging system.

[0214] In various cases, the receiver 1208 may be mounted on a trocar or cannula (such as trocar 1210b), for example, through which the surgical device 1202b is positioned. In other cases, the receiver 1208 may be mounted on a separate robotic arm whose three-dimensional position is known. In various cases, the receiver 1208 may be mounted on a movable arm separate from the robot that controls the surgical device 1202a, or may be mounted to an operating room (OR) table that can be registered with the robot's coordinate plane during surgery. In such cases, the positions of the transmitter 1206 and the receiver 1208 may be registered with the same coordinate plane, so that distances can be triangulated based on the output from the time-of-flight sensor system 1204.

[0215] Combining a time-of-flight sensor system and near-infrared spectroscopy (NIRS), known as TOF-NIRS, which is capable of measuring time-resolved signatures of NIR light with nanosecond resolution, is described in an article entitled “TIME-OF-FLIGHT NEAR-INFRAREDSPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY INGRAPEFRUIT” (Journal of the American Society for Horticultural Science, May 2013, Vol. 138, No. 3, pp. 225-228), which is incorporated herein by reference in its entirety and is available at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0216] In various cases, the time-of-flight spectral waveform is configured to determine the depth of a critical structure and / or the proximity of a surgical device to a critical structure. In addition, various surgical visualization systems disclosed herein include a surface mapping logic component that is configured to create a three-dimensional rendering of the surface of visible tissue. In such cases, the clinician can be aware of the proximity (or lack thereof) of the surgical device to the critical structure even when visible tissue obstructs the critical structure. In one case, the topography of the surgical site is provided on a monitor by the surface mapping logic component. If the critical structure is close to the surface of the tissue, spectral imaging can convey the location of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other cases, spectral imaging can detect structures 10 mm or 20 mm below the surface of the tissue. Based on the known limitations of the spectral imaging system, the system is configured to convey that the critical structure is out of range even if the spectral imaging system cannot detect the critical structure at all. Therefore, the clinician can continue to move the surgical device and / or manipulate the tissue. When the critical structure moves into the range of the spectral imaging system, the system can identify the structure and therefore convey that the structure is within range. In such cases, an alert can be provided when a structure is initially identified and / or further moved within a predefined proximity zone. In such cases, proximity information (i.e., lack of proximity) can be provided to the clinician even if the spectral imaging system does not identify a critical structure using known boundaries / ranges.

[0217] Various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures during surgery and alert clinicians before damaging critical structures through accidental dissection and / or transection. In various aspects, the surgical visualization system is configured to identify, for example, one or more of the following critical structures: ureters, intestines, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], sacral promontory facial nerve, vagus nerve and their branches), blood vessels (including the pulmonary artery and lobar arteries and pulmonary veins and lobar veins, the inferior mesenteric artery [IMA] and its branches, the superior rectal artery, the sigmoid artery and the left colic artery), the superior mesenteric artery (SMA) and its branches (including the middle colic artery, the right colic artery, the ileal artery), the hepatic artery and its branches, the portal vein and its branches, the splenic artery / vein and its branches, the external and internal (lower abdominal) ileal vessels, the short gastric arteries, the uterine arteries, the middle sacral vessels, and lymph nodes. In addition, the surgical visualization system is configured to indicate the proximity of a surgical device to a critical structure and / or alert clinicians when a surgical device is in proximity to a critical structure.

[0218] Various aspects of the present disclosure provide for intraoperative critical structure identification (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, which enable visualization of critical structures, such as those below the surface of tissue (e.g., 1.0 cm to 1.5 cm below the surface of the tissue). In other cases, the surgical visualization system may identify structures less than 1.0 cm or greater than 1.5 cm below the surface of the tissue. For example, a surgical visualization system that can identify structures that are only within 0.2 mm of the surface may be valuable if the structure is not otherwise visible due to depth. In various aspects, the surgical visualization system may enhance the clinician's view, for example, by utilizing a virtual depiction of the critical structure as a visible white light image superimposed on the surface of the visible tissue. The surgical visualization system may provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and may, for example, provide a proximity alert when the distal tip of the surgical instrument moves within a specific range of the critical structure (e.g., within 1.0 cm of the critical structure).

[0219] Various surgical visualization systems disclosed herein can identify when dissection is too close to a critical structure. Dissection may be "too close" to a critical structure based on temperature (i.e., too much heat near a critical structure that could risk damaging / heating / melting the critical structure) and / or tension (i.e., too much tension near a critical structure that could risk damaging / tearing / pulling the critical structure). For example, such surgical visualization systems can facilitate dissection around a vessel when skeletonizing the vessel prior to ligation. In various cases, a thermal imaging camera can be utilized to read the heat at the surgical site and provide a warning to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the tool temperature is above a predefined threshold (such as, for example, 120°F), a warning can be provided to the clinician at a first distance (such as, for example, 10 mm), and if the tool temperature is less than or equal to the predefined threshold, a warning can be provided to the clinician at a second distance (such as, for example, 5 mm). The predefined thresholds and / or warning distances can be default settings and / or programmable by the clinician. Additionally or alternatively, the proximity alert may be associated with a thermal measurement made by the tool itself, such as a thermocouple measuring heat in the distal jaws of, for example, a monopolar or bipolar dissector or vessel sealer.

[0220] The various surgical visualization systems disclosed herein can provide sufficient sensitivity with respect to critical structures and specificity to enable clinicians to confidently perform rapid but safe dissections based on standards of care and / or device safety data. The systems can function in real time during surgical procedures intraoperatively with minimal, and in various cases, no, ionizing radiation risk to the patient or clinician. In contrast, during fluoroscopy, the patient and clinician can be exposed to ionizing radiation via, for example, an X-ray beam used to view anatomical structures in real time.

[0221] The various surgical visualization systems disclosed herein can be configured to, for example, detect and identify one or more desired types of critical structures in a forward path of a surgical device, such as when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system can be configured to detect and identify one or more desired types of critical structures, for example, in the surrounding area of the surgical device and / or in multiple planes / dimensions.

[0222] The various surgical visualization systems disclosed herein can be easy to operate and / or interpret. Furthermore, the various surgical visualization systems can incorporate an "override" feature that allows a clinician to override default settings and / or operations. For example, a clinician can selectively disable alerts from the surgical visualization system and / or move closer to a critical structure than the surgical visualization system suggests, such as when the risk to the critical structure is less than the risk of avoiding the area (e.g., when removing cancer surrounding a critical structure, the risk of leaving cancerous tissue behind may be greater than the risk of causing damage to the critical structure).

[0223] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, the specific implementation of a surgical visualization system may not change the way surgical procedures are performed. Furthermore, a surgical visualization system may be more economical compared to the cost of accidental transections. Data suggests that reduced accidental damage to critical structures can drive incremental compensation.

[0224] The various surgical visualization systems disclosed herein can operate in real time or near real time and far enough in advance to enable the clinician to anticipate critical structures. For example, the surgical visualization system can provide sufficient time to "slow down, assess, and avoid" in order to maximize the efficiency of the surgical procedure.

[0225] The various surgical visualization systems disclosed herein may not require contrast agents or dyes to be injected into tissue. For example, spectral imaging can be configured to visualize hidden structures during surgery without the use of contrast agents or dyes. In other cases, contrast agents can be more easily injected into the appropriate tissue layers than with other visualization systems. For example, the time between contrast agent injection and visualization of critical structures can be less than two hours.

[0226] The various surgical visualization systems disclosed herein can be associated with clinical data and / or device data. For example, the data can provide boundaries regarding how close an energy-enabled surgical device (or other potentially damaging device) should be to tissue that the surgeon does not want to damage. Any data modules that interact with the surgical visualization systems disclosed herein can be provided integrally with or separately from a robot to enable use with standalone surgical devices, such as in open or laparoscopic surgery. In various cases, the surgical visualization system can be compatible with a robotic surgical system. For example, visualization images / information can be displayed on a robotic console.

[0227] In various cases, the clinician may not know the position of the critical structure relative to the surgical tool. For example, when the critical structure is embedded in the tissue, the clinician may not be able to determine the position of the critical structure. In some cases, the clinician may want to keep the surgical device outside the range of the orientation around the critical structure and / or away from the visible tissue covering the hidden critical structure. When the orientation of the hidden critical structure is unknown, the clinician may have the risk of moving too close to the critical structure, which may cause unintentional trauma to the critical structure and / or dissection and / or excessive energy, heat and / or tension near the critical structure. Alternatively, the clinician may keep too far away from the suspected position of the critical structure and have the risk of affecting the tissue in an attempt to avoid the critical structure at a less ideal position.

[0228] The present invention provides a surgical visualization system that presents tracking of a surgical device relative to one or more critical structures. For example, the surgical visualization system can track the proximity of a surgical device relative to a critical structure. Such tracking can occur intraoperatively, in real time, and / or near real time. In various cases, the tracking data can be provided to a clinician via a display screen (e.g., a monitor) of an imaging system.

[0229] In one aspect of the present disclosure, a surgical visualization system includes: a surgical device including an emitter configured to emit a structured light pattern onto a visible surface; an imaging system including a camera configured to detect the structured light pattern on an embedded structure and the visible surface; and control circuitry in signal communication with the camera and the imaging system, wherein the control circuitry is configured to determine a distance from the surgical device to the embedded structure and provide a signal indicating the distance to the imaging system. For example, the distance can be determined by calculating the distance from the camera to a critical structure illuminated using fluoroscopy and based on a three-dimensional view of the illuminated structure provided by images from multiple lenses of the camera (e.g., a left lens and a right lens). For example, the distance from the surgical device to the critical structure can be triangulated based on the known positions of the surgical device and the camera. Alternative devices for determining the distance to the embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be employed. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue that overlays / covers the embedded critical structure. For example, the surgical visualization system can identify and enhance the view of hidden critical structures by drawing a schematic representation of the hidden critical structures on the visible structures (such as a line on the surface of the visible tissue). The surgical visualization system can also determine the distance to the enhanced line on the visible tissue.

[0230] By providing clinicians with up-to-date information about the proximity of surgical devices to hidden critical structures and / or visible structures, as provided by various surgical visualization systems disclosed herein, clinicians can make more informed decisions regarding the placement of surgical devices relative to hidden critical structures. For example, clinicians can view the distance between surgical devices and critical structures in real time / intraoperatively, and in some cases, alerts and / or warnings can be provided by the imaging system when the surgical device moves within a predefined proximity and / or zone of a critical structure. In some cases, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a "no-fly" zone near a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm, or more of the critical structure). In such cases, clinicians can maintain momentum throughout the surgical procedure without having to monitor the suspected location of critical structures and the proximity of the surgical device thereto. As a result, certain surgical procedures can be performed faster, with fewer pauses / interruptions, and / or with improved accuracy and / or certainty. In one aspect, surgical visualization systems can be used to detect tissue variability, such as variability of tissue within an organ, to differentiate between tumors / cancer tissue / unhealthy tissue and healthy tissue. Such surgical visualization systems can maximize the removal of unhealthy tissue while minimizing the removal of healthy tissue.

[0231] Surgical Hub System

[0232] The various visualization or imaging systems described herein can be incorporated into a surgical hub system, such as in conjunction with Figure 17-19 are shown and described in further detail below.

[0233] See also Figure 17 , a computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include a remote server 2113. In one example, Figure 17 , the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and handheld intelligent surgical instruments 2112, which are configured to communicate with each other and / or with a hub 2106. In some aspects, the surgical system 2102 may include M number of hubs 2106, N number of visualization systems 2108, O number of robotic systems 2110, and P number of handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to one.

[0234] Figure 18 An example of a surgical system 2102 is shown for performing a surgical procedure on a patient lying flat on an operating table 2114 in a surgical operating room 2116. A robotic system 2110 is used as part of the surgical system 2102 during the surgical procedure. The robotic system 2110 includes a surgeon's console 2118, a patient-side cart 2120 (surgical robot), and a surgical robotic hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side cart 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision in the patient's body. Images of the surgical site can be obtained through a medical imaging device 2124, which can be manipulated by the patient-side cart 2120 to orient the imaging device 2124. The robotic hub 2122 can be used to process the images of the surgical site for subsequent display to the surgeon through the surgeon's console 2118.

[0235] Other types of robotic systems can be readily adapted for use with the surgical system 2102. Various examples of robotic systems and surgical tools suitable for use with the present disclosure are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0236] Various examples of cloud-based analytics performed by the cloud 2104 and suitable for use in the present disclosure are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0237] In various aspects, imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge coupled device (CCD) sensors and complementary metal oxide semiconductor (CMOS) sensors.

[0238] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0239] One or more illumination sources can be configured to radiate electromagnetic energy in the visible spectrum as well as in the invisible spectrum. The visible spectrum (sometimes referred to as the optical spectrum or the luminescence spectrum) is that portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye and can be referred to as visible light or simply light. A typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0240] The invisible spectrum (i.e., the non-luminescent spectrum) is the portion of the electromagnetic spectrum below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, and they become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, and they become invisible ultraviolet, x-ray, and gamma-ray electromagnetic radiation.

[0241] In various aspects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use in the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and hysteroscopes.

[0242] In one aspect, an imaging device employs multispectral monitoring to discern topography and underlying structure. A multispectral image is an image that captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that are sensitive to specific wavelengths, including light from frequencies outside the visible range, such as IR and ultraviolet. Spectral imaging can allow for the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated by reference into this disclosure. Multispectral monitoring can be a useful tool for repositioning the surgical field after completing the surgical task to perform one or more of the previously described tests on the treated tissue.

[0243] It is self-evident that during any surgery, the operating room and surgical equipment need to be strictly sterilized. The strict hygiene and sterilization conditions required in a "surgical room" (i.e., an operating room or treatment room) require the highest possible sterility of all medical devices and equipment. Part of this sterilization process is the need to sterilize any material that contacts the patient or penetrates the sterile field, including the imaging device 2124 and its attachments and components. It should be understood that the sterile field can be considered to be a designated area that is considered to be free of microorganisms, such as in a tray or in a sterile towel, or the sterile field can be considered to be the area around the patient that has been prepared for a surgical procedure. The sterile field can include appropriately dressed, scrubbed team members, and all equipment and fixtures in the area.

[0244] In various aspects, the visualization system 2108 includes one or more imaging sensors, one or more image processing units, one or more storage arrays, and one or more displays strategically positioned relative to the sterile field, such as Figure 18 In one aspect, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. Various components of the visualization system 2108 are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0245] like Figure 18 As shown in FIG, a primary display 2119 is positioned within the sterile field so as to be visible to an operator at the operating table 2114. Additionally, a visualization tower 21121 is positioned outside the sterile field. Visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109, facing away from each other. A visualization system 2108, directed by hub 2106, is configured to coordinate information flow to operators inside and outside the sterile field using displays 2107, 2109, and 2119. For example, hub 2106 can cause visualization system 2108 to display a snapshot of the surgical site recorded by imaging device 2124 on non-sterile display 2107 or 2109, while simultaneously maintaining a real-time feed of the surgical site on primary display 2119. The snapshots on non-sterile display 2107 or 2109 can allow a non-sterile operator to, for example, perform diagnostic procedures related to a surgical procedure.

[0246] In one aspect, the hub 2106 is further configured to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 to the primary display 2119 within the sterile field, where it can be viewed by a sterile operator at the operating table. In one example, the input can be a modified form of a snapshot displayed on the non-sterile display 2107 or 2109, which can be routed to the primary display 2119 via the hub 2106.

[0247] See also Figure 18 , surgical instruments 2112 are used during a surgical procedure as part of the surgical system 2102. The hub 2106 is also configured to coordinate the flow of information to the displays of the surgical instruments 2112, as described in various U.S. patent applications, which are incorporated by reference into this disclosure. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 can be routed by the hub 2106 to the surgical instrument display 2115 within the sterile field, where the operator of the surgical instrument 2112 can observe the input or feedback. Exemplary surgical instruments suitable for use with the surgical system 2102 are described in various U.S. patent applications, which are incorporated by reference into this disclosure.

[0248] Figure 19 A computer-implemented interactive surgical system 2200 is shown. The computer-implemented interactive surgical system 2200 is similar in many respects to the computer-implemented interactive surgical system 2100. The surgical system 2200 includes at least one surgical hub 2236 in communication with a cloud 2204, which may include a remote server 2213. In one aspect, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to a plurality of operating room devices, such as, for example, smart surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. As Figure 19 As shown in the example of FIG, a surgical hub 2236 is coupled to an imaging module 2238 (which is coupled to an endoscope 2239), a generator module 2240 coupled to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communications module 2230, a processor module 2232, a storage array 2234, smart devices / instruments 2235 optionally coupled to a display 2237, and a contactless sensor module 2242. Operating room devices are coupled to cloud computing resources and data storage via the surgical hub 2236. The robotic hub 2222 can also be connected to the surgical hub 2236 and cloud computing resources. Devices / instruments 2235, visualization system 2209, etc. can be coupled to the surgical hub 2236 via wired or wireless communication standards or protocols, as described herein. The surgical hub 2236 can be coupled to a hub display 2215 (e.g., monitor, screen) to display and overlay images received from imaging modules, device / instrument displays, and / or other visualization systems 208. The hub display can also display data received from devices connected to the modular control tower in conjunction with images and overlays.

[0249] Situational Awareness

[0250] The various visualization systems or aspects of the visualization systems described herein may be used as part of a situational awareness system that may be provided by the surgical hub 2106, 2236 ( Figure 17-19 ) to be implemented or executed. In particular, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and motion), tissue, structures, users, and other objects located within the surgical field or operating room can provide contextual data that the situational awareness system can utilize to infer the type of surgical procedure being performed or a step thereof, the type of tissue and / or structure being manipulated by the surgeon, and the like. The situational awareness system can then utilize this contextual data to provide alerts to the user, advise the user on subsequent steps or actions, prepare the surgical device for use (e.g., activating an electrosurgical generator in anticipation of using an electrosurgical instrument in a subsequent step of the surgical procedure), intelligently control the surgical instrument (e.g., customizing surgical instrument operating parameters based on the specific health status of each patient), and the like.

[0251] While "smart" devices that include control algorithms that respond to sensed data may be an improvement over "dumb" devices that operate without regard to the sensed data, some sensed data may be incomplete or uncertain when considered in isolation, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the surgical context (e.g., knowing the type of tissue being operated on or the type of surgery being performed), the control algorithm may incorrectly or suboptimally control the modular device given specific context-free sensed data. The modular device may include any surgical device that can be controlled by the situational awareness system, such as a visualization system device (e.g., a camera or display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a smoke evacuator). For example, the optimal approach for controlling a control algorithm for a surgical instrument in response to a particular sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., tear resistance) and therefore respond differently to actions taken by the surgical instrument. Therefore, it may be desirable for the surgical instrument to take different actions even when sensing the same measurement value for a particular parameter. As a specific example, the optimal way to control a surgical stapling and cutting instrument in response to the instrument sensing an unexpectedly high force for closing its end effector will vary depending on whether the tissue type is prone to tearing or resistant to tearing. For tissue that is prone to tearing (such as lung tissue), the instrument's control algorithm will optimally gradually reduce the motor speed in response to the unexpectedly high force for closing, thereby avoiding tearing the tissue. For tissue that is resistant to tearing (such as gastric tissue), the instrument's control algorithm will optimally gradually increase the motor speed in response to the unexpectedly high force for closing, thereby ensuring that the end effector is properly clamped on the tissue. Without knowing whether lung tissue or gastric tissue has been clamped, the control algorithm may make suboptimal decisions.

[0252] One solution utilizes a surgical hub that includes a system configured to derive information about the surgical protocol being performed based on data received from various data sources, and then control paired modular devices accordingly. In other words, the surgical hub is configured to infer information about the surgical protocol from the received data, and then control the modular devices paired with the surgical hub based on the context of the inferred surgical protocol. Figure 20A diagram of a situational awareness surgical system 2400 according to at least one aspect of the present disclosure is shown. In some examples, data sources 2426 include, for example, modular apparatus 2402 (which may include sensors configured to detect parameters associated with a patient and / or the modular apparatus itself), database 2422 (e.g., an EMR database containing patient records), and patient monitoring devices 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0253] The surgical hub 2404 (which may be similar to the hub 106 in many respects) may be configured to be able to derive contextual information related to the surgical procedure from the data, for example, based on a particular combination of the received data or a particular order in which the data is received from the data source 2426. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity that is the subject of the procedure. Some aspects of the surgical hub 2404's ability to derive or infer information related to the surgical procedure from the received data may be referred to as "situational awareness." In one example, the surgical hub 2404 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 2404 that derives contextual information related to the surgical procedure from the received data.

[0254] The situational awareness system of the surgical hub 2404 can be configured to derive contextual information from data received from the data source 2426 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or a machine learning system (e.g., an artificial neural network) that has been trained on training data to associate various inputs (e.g., data from the database 2422, the patient monitoring device 2424, and / or the modular devices 2402) with corresponding contextual information about the surgical procedure. In other words, the machine learning system can be trained to accurately derive contextual information about the surgical procedure from the provided inputs. In another example, the situational awareness system can include a lookup table that stores pre-characterized contextual information about the surgical procedure associated with one or more inputs (or input ranges) corresponding to the contextual information. In response to a query utilizing the one or more inputs, the lookup table can return corresponding contextual information used by the situational awareness system to control the modular devices 2402. In one example, the contextual information received by the situational awareness system of the surgical hub 2404 is associated with a specific control adjustment or set of control adjustments for one or more modular devices 2402. In another example, the situational awareness system includes an additional machine learning system, lookup table, or other such system that generates or retrieves one or more control adjustments for one or more modular devices 2402 when provided with contextual information as input.

[0255] The surgical hub 2404 incorporating a situational awareness system provides numerous benefits to the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which in turn improves processing accuracy and / or use of the data during the surgical procedure. Returning to the previous example, the situational awareness surgical hub 2404 can determine the type of tissue being operated on; thus, when an unexpectedly high force is detected on the end effector of a closing surgical instrument, the situational awareness surgical hub 2404 can appropriately ramp up or ramp down the motor speed of the surgical instrument appropriate for the tissue type.

[0256] As another example, the type of tissue being operated on can affect the adjustment of the compression rate and load threshold of the surgical stapling and cutting instrument for a specific tissue gap measurement value. The situational awareness surgical hub 2404 can infer whether the surgical procedure being performed is thoracic surgery or abdominal surgery, thereby allowing the surgical hub 2404 to determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression rate and load threshold of the surgical stapling and cutting instrument for the type of tissue.

[0257] As yet another example, the type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situational awareness surgical hub 2404 can determine whether the surgical site is under pressure (by determining that the surgical procedure is utilizing insufflation) and determine the type of surgery. Since one type of surgery is typically performed in a specific body cavity, the surgical hub 2404 can then control the motor speed of the smoke evacuator appropriately for the body cavity in which the procedure is being performed. Thus, the situational awareness surgical hub 2404 can provide consistent smoke evacuation for both thoracic and abdominal surgeries.

[0258] As yet another example, the type of procedure being performed may affect the optimal energy level at which an ultrasonic surgical instrument or radio frequency (RF) electrosurgical instrument operates. For example, arthroscopic procedures require higher energy levels because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is immersed in fluid. The situational awareness surgical hub 2404 may determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 may then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on may affect the optimal energy level at which an ultrasonic surgical instrument or RF electrosurgical instrument operates. The situational awareness surgical hub 2404 may determine the type of surgical procedure being performed and then customize the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument, respectively, based on the expected tissue profile of the surgical procedure. In addition, the situational awareness surgical hub 2404 may be configured to be able to adjust the energy level of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the entire surgical procedure, rather than just on a procedure-by-procedure basis. The situational awareness surgical hub 2404 can determine the step of a surgical procedure being performed or to be subsequently performed and then update the control algorithm for the generator and / or ultrasonic surgical instrument or RF electrosurgical instrument to set the energy level at a value appropriate for the expected tissue type based on the surgical step.

[0259] As yet another example, data may be extracted from additional data sources 2426 to improve conclusions extracted by the surgical hub 2404 from one data source 2426. The situational awareness surgical hub 2404 may augment the data it receives from the modular devices 2402 with contextual information about the surgical procedure that has been constructed from other data sources 2426. For example, the situational awareness surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) based on video or image data received from a medical imaging device. However, in some cases, the video or image data may be inconclusive. Thus, in one example, the surgical hub 2404 may be further configured to combine physiological measurements (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 communicatively coupled to the surgical hub 2404). Figure 2 )) to determine the integrity of the suture or tissue weld. In other words, the situational awareness system of the surgical hub 2404 can consider physiological measurement data to provide additional context when analyzing visualization data. Additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0260] Another benefit includes actively and automatically controlling the paired modular devices 2402 according to the specific steps of the surgical procedure being performed, thereby reducing the number of times medical personnel need to interact with or control the surgical system 2400 during the surgical procedure. For example, if the situational awareness surgical hub 2404 determines that a subsequent step in the procedure requires the use of an RF electrosurgical instrument, it can actively activate the generator connected to the instrument. Actively activating the energy source allows the instrument to be ready for use as soon as the previous step of the procedure is completed.

[0261] As another example, the situational awareness surgical hub 2404 may determine whether the current or subsequent steps of the surgical procedure require a different view or degree of magnification on the display based on the feature(s) that the surgeon anticipates needing to view at the surgical site. The surgical hub 2404 may then proactively change the displayed view (e.g., provided by a medical imaging device for the visualization system 108) accordingly, such that the display is automatically adjusted throughout the surgical procedure.

[0262] As yet another example, the situational awareness surgical hub 2404 can determine which step of a surgical procedure is currently being performed or will subsequently be performed and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 2404 can be configured to automatically call up data screens based on the step of the surgical procedure being performed, without having to wait for the surgeon to request that specific information.

[0263] Another benefit includes checking for errors during the setup of a surgical procedure or during the course of a surgical procedure. For example, the situational awareness surgical hub 2404 can determine whether the operating room is correctly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve the corresponding list, product location, or setup requirements (e.g., from memory), and then compare the current operating room layout with the standard layout determined by the surgical hub 2404 for the type of surgical procedure being performed. In one example, the surgical hub 2404 can be configured to compare a list of items for surgery (e.g., scanned by a suitable scanner) and / or a list of devices paired with the surgical hub 2404 with a recommended or expected list of items and / or devices for a given surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing if there is any discontinuity between the lists. In one example, the surgical hub 2404 can be configured to determine the relative distance or position of the modular device 2402 and the patient monitoring device 2424, for example, via a proximity sensor. The surgical hub 2404 can compare the relative position of the devices to a recommended or expected layout for a particular surgical procedure. The surgical hub 2404 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout if there is any discontinuity between the layouts.

[0264] As another example, the situational awareness surgical hub 2404 can determine whether a surgeon (or other medical personnel) is making an error or otherwise deviating from an expected course of action during a surgical procedure. For example, the surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve a corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the surgical procedure to the expected steps or equipment determined by the surgical hub 2404 for the type of surgical procedure being performed. In one example, the surgical hub 2404 can be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0265] In general, the situational awareness system for the surgical hub 2404 improves surgical outcomes by adjusting surgical instruments (and other modular devices 2402) to the specific context of each surgical procedure (such as adjusting for different tissue types) and validating movements during the surgical procedure. The situational awareness system also improves the surgeon's efficiency in performing surgical procedures by automatically suggesting next steps, providing data, and adjusting displays and other modular devices 2402 in the operating room based on the specific context of the surgery.

[0266] Now see Figure 21 , which shows a depiction of a hub such as surgical hub 106 or 206 ( Figures 1 to 11 ). Timeline 2500 is an illustrative surgical procedure and contextual information that the surgical hub 106, 206 can derive from data received from data sources at each step in the surgical procedure. Timeline 2500 depicts typical steps that nurses, surgeons, and other medical staff would take during a segmentectomy surgery, starting with setting up the operating room and ending with transferring the patient to the postoperative recovery room.

[0267] The situational awareness surgical hub 106, 206 receives data from data sources throughout the surgical procedure, including data generated each time medical personnel utilize a modular device paired with the surgical hub 106, 206. The surgical hub 106, 206 can receive this data from the paired modular devices and other data sources and continuously derive inferences about the ongoing surgery (i.e., contextual information) as new data is received, such as which step of the surgery is being performed at any given time. The situational awareness system of the surgical hub 106, 206 is capable of, for example, recording data related to the procedure for generating reports, verifying the steps being taken by medical personnel, providing data or prompts that may be related to specific procedure steps (e.g., via a display screen), adjusting modular devices based on the context (e.g., activating a monitor, adjusting the field of view (FOV) of a medical imaging device, or changing the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument), and taking any other such actions described above.

[0268] As a first step 2502 in this exemplary operation, hospital staff retrieves the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hub 106, 206 determines that the operation to be performed is a thoracic operation.

[0269] In a second step 2504, staff scans the incoming medical supplies for the procedure. The surgical hub 106, 206 cross-references the scanned supplies with a list of supplies utilized in various types of surgeries and confirms that the mix of supplies corresponds to a thoracic procedure. Additionally, the surgical hub 106, 206 can also determine that the procedure is not a wedge procedure (because the incoming supplies lack certain supplies required for a thoracic wedge procedure or are otherwise not consistent with a thoracic wedge procedure).

[0270] In a third step 2506, the medical personnel scans the patient belt via a scanner communicatively connected to the surgical hub 106, 206. The surgical hub 106, 206 may then confirm the identity of the patient based on the scanned data.

[0271] In a fourth step 2508, the medical staff turns on the auxiliary equipment. The auxiliary equipment utilized may vary depending on the type of surgical procedure and the techniques to be used by the surgeon, but in this exemplary case, they include a smoke evacuator, an insufflator, and a medical imaging device. When activated, as part of their initialization process, the auxiliary equipment, which is a modular device, may automatically pair with a surgical hub 106, 206 located in a specific vicinity of the modular device. The surgical hub 106, 206 may then derive contextual information about the surgical procedure by detecting the type of modular device with which it was paired during this preoperative or initialization phase. In this specific example, the surgical hub 106, 206 determines that the surgical procedure is a VATS procedure based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, the list of medical supplies used in the procedure, and the type of modular device connected to the hub, the surgical hub 106, 206 can generally infer the specific procedure that the surgical team will perform. Once the surgical hub 106, 206 knows what specific procedure is being performed, the surgical hub 106, 206 can retrieve the steps of that procedure from memory or the cloud and then cross-reference the data it subsequently receives from connected data sources (e.g., modular devices and patient monitoring devices) to infer what steps of the surgical procedure the surgical team is performing.

[0272] In step 510, the staff attaches EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hub 106, 206. When the surgical hub 106, 206 begins receiving data from the patient monitoring devices, the surgical hub 106, 206 thus confirms that the patient is in the operating room.

[0273] In step 2512, medical personnel induce anesthesia in the patient. The surgical hub 106, 206 may infer that the patient is under anesthesia based on data from the modular device and / or the patient's monitoring device (including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof). Upon completion of step 2512, the preoperative portion of the lung segmentectomy surgery is complete, and the surgical portion begins.

[0274] In step 2514, the patient's lung being operated on is collapsed (while ventilation is switched to the contralateral lung). For example, the surgical hub 106, 206 may infer from ventilator data that the patient's lung has collapsed. The surgical hub 106, 206 may infer that the surgical portion of the procedure has begun because it may compare the detection of the patient's lung collapsing with the expected steps of the procedure (which may have been previously accessed or retrieved) and determine that collapsing the lung is the first surgical step in this particular procedure.

[0275] In step eight 2516, a medical imaging device (e.g., an endoscope) is inserted and video from the medical imaging device is started. The surgical hub 106, 206 receives the medical imaging device data (i.e., video or image data) through its connection to the medical imaging device. After receiving the medical imaging device data, the surgical hub 106, 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 106, 206 can determine that the particular procedure being performed is a segmentectomy, rather than a lobectomy (note that wedge procedures have been excluded based on the data received by the surgical hub 106, 206 at step two 2504 of the procedure). From the medical imaging device 124 ( Figure 2) can be used to determine contextual information related to the type of procedure being performed in a number of different ways, including by determining the angle of the medical imaging device's visualization orientation relative to the patient's anatomy, monitoring the number of medical imaging devices utilized (i.e., activated and paired with the surgical hub 106, 206), and monitoring the type of visualization devices utilized. For example, one technique for performing a VATS lobectomy places a camera in the lower anterior corner of the patient's thorax above the diaphragm, while a technique for performing a VATS segmentectomy places a camera in an anterior intercostal position relative to the segmental fissure. For example, using pattern recognition or machine learning techniques, a situational awareness system can be trained to recognize the positioning of the medical imaging device based on visualization of the patient's anatomy. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As another example, a technique for performing a VATS segmentectomy utilizes an infrared light source (which can be communicatively coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure that is not used in a VATS pulmonary resection. By tracking any or all of this data from the medical imaging device, the surgical hub 106, 206 can therefore determine the specific type of surgical procedure being performed and / or the techniques used for a particular type of surgical procedure.

[0276] In step 2518, the surgical team begins the dissection step of the procedure. The surgical hub 106, 206 may infer that the surgeon is dissecting to mobilize the patient's lung because it receives data from the RF generator or ultrasonic generator indicating that an energy instrument is being fired. The surgical hub 106, 206 may intersect the received data with the search steps of the surgical procedure to determine that the energy instrument fired at this point in the procedure (i.e., after the previously discussed surgical steps have been completed) corresponds to the dissection step. In some cases, the energy instrument may be an energy tool mounted to a robotic arm of a robotic surgical system.

[0277] In step 10 2520 , the surgical team proceeds with the ligation step of the procedure. The surgical hub 106 , 206 can infer that the surgeon is ligating the artery and vein because it receives data from the surgical stapling and cutting instrument indicating that the instrument is being fired. Similar to the previous step, the surgical hub 106 , 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the retrieval step in the process. In some cases, the surgical instrument can be a surgical tool mounted to a robotic arm of a robotic surgical system.

[0278] In step 11 2522, the segmentectomy portion of the surgery is performed. The surgical hub 106, 206 can infer that the surgeon is transecting soft tissue based on data from the surgical stapling and cutting instrument (including data from its cartridge). The cartridge data can correspond to, for example, the size or type of staples fired by the instrument. Because different types of staples are used for different types of tissue, the cartridge data can indicate the type of tissue being sutured and / or transected. In this case, the type of staple fired is for soft tissue (or other similar tissue types), which allows the surgical hub 106, 206 to infer that the segmentectomy portion of the surgery is being performed.

[0279] In step 2524, the node dissection step is performed. Based on data received from the generator indicating that an RF or ultrasonic instrument is being fired, the surgical hub 106, 206 can infer that the surgical team is dissecting a node and performing a leak test. For this particular procedure, the RF or ultrasonic instrument utilized after transecting the soft tissue corresponds to the node dissection step, which allows the surgical hub 106, 206 to make such an inference. It should be noted that surgeons regularly switch back and forth between surgical stapling / cutting instruments and surgical energy (i.e., RF or ultrasonic) instruments depending on the specific steps in the procedure, as different instruments are better suited for specific tasks. Therefore, the specific sequence in which the stapling / cutting instruments and surgical energy instruments are used can indicate the steps of the procedure being performed by the surgeon. Furthermore, in some cases, robotic tools may be used for one or more steps in the surgical procedure, and / or handheld surgical instruments may be used for one or more steps in the surgical procedure. One or more surgeons may, for example, alternate between robotic tools and handheld surgical instruments and / or may use the devices simultaneously. Upon completion of step 2524, the incision is closed and the postoperative portion of the procedure begins.

[0280] Thirteenth step 2526, reverse anesthesia to the patient. For example, the surgical hub 106, 206 may infer that the patient is waking up from anesthesia based on, for example, ventilator data (ie, the patient's breathing rate begins to increase).

[0281] Finally, in step 14 2528, the medical staff removes the various patient monitoring devices from the patient. Thus, when the hub loses EKG, BP, and other data from the patient monitoring devices, the surgical hub 2106, 2236 can infer that the patient is being transferred to a recovery room. As can be seen from the description of this exemplary surgery, the surgical hub 2106, 2236 can determine or infer when each step of a given surgical procedure occurred based on data received from various data sources communicatively coupled to the surgical hub 2106, 2236.

[0282] Situational awareness is further described in various U.S. patent applications, which are incorporated by reference into this disclosure, which is incorporated by reference herein. In some cases, the operation of a robotic surgical system (including, for example, the various robotic surgical systems disclosed herein) may be controlled by the hub 2106, 2236 based on its situational awareness and / or feedback from its components and / or based on information from the cloud 2104 ( Figure 17 ) information to control.

[0283] Figure 22 4000 is a logic flow diagram of a process 4000 according to at least one aspect of the present disclosure, the logic flow diagram depicting a control program or logic configuration for associating visualization data with instrument data. The process 4000 is typically performed during a surgical procedure and includes receiving or deriving 4001 a first data set, visualization data, indicating visual aspects of a surgical instrument relative to a surgical field of view from a surgical visualization system, receiving or deriving 4002 a second data set, instrument data, indicating functional aspects of the surgical instrument from the surgical instrument, and associating 4003 the first data set with the second data set.

[0284] In at least one example, associating the visualization data with the instrument data is accomplished by developing a composite dataset from the visualization data and the instrument data. Process 4000 can further include comparing the composite dataset with another composite dataset, which can be received from an external source and / or derived from a previously collected composite dataset. In at least one example, process 4000 includes displaying a comparison of the two composite datasets, as described in more detail below.

[0285] The visualization data of process 4000 may indicate visual aspects of the end effector of the surgical instrument relative to tissue in the surgical field of view. Additionally or alternatively, the visualization data may indicate visual aspects of tissue treated by the end effector of the surgical instrument. In at least one example, the visualization data represents one or more positions of the end effector or a component thereof relative to the tissue in the surgical field of view. Additionally or alternatively, the visualization data may represent one or more movements of the end effector or a component thereof relative to the tissue in the surgical field of view. In at least one example, the visualization data represents one or more changes in the shape, size, and / or color of tissue treated by the end effector of the surgical instrument.

[0286] In various aspects, visualization data is derived from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108). The visualization data may be derived from various measurements, readings, and / or any other suitable parameters monitored and / or captured by the surgical visualization system, such as in combination with Figures 1 to 18In various examples, the visualization data indicates one or more visual aspects of tissue in the surgical field of view and / or one or more visual aspects of the surgical instrument relative to tissue in the surgical field of view. In some examples, the visualization data represents or identifies the position of the end effector of the surgical instrument relative to key structures in the surgical field of view (e.g., Figure 1 In some examples, the visualization data is derived from surface mapping data, imaging data, tissue identification data, and / or distance data calculated by surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141.

[0287] In at least one example, visualization data is derived from tissue recognition and geometric surface mapping performed by visualization system 100 in conjunction with distance sensor system 104, such as in conjunction with Figure 1 In at least one example, the visualization data is derived from measurements, readings, or any other sensor data captured by the imaging device 120. Figure 1 As described, the imaging device 120 is a spectral camera (eg, a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of an image based on molecular responses to different wavelengths.

[0288] Additionally or alternatively, the visualization data may be derived from measurements, readings, or any suitable sensor data captured by any suitable imaging device, including, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In at least one example, the visualization data is derived from a visualization system 160 that includes an optical waveform emitter 123 and a waveform sensor 122 that is configured to detect reflected waveforms, such as in combination with Figures 3 and 4 and Figures 13 to Figure 16 In yet another example, the visualization data is derived from a visualization system that includes a three-dimensional (3D) camera and associated electronic processing circuitry, such as visualization system 500. In yet another example, the visualization data is derived from a structured (or patterned) light system 700 that combines Figure 12 The foregoing examples may be used individually or in combination to derive visualization data for process 4000 .

[0289] The instrument data of process 4000 may indicate one or more operations of one or more internal components of a surgical instrument. In at least one example, the instrument data represents one or more operating parameters of the internal components of the surgical instrument. The instrument data may represent one or more positions and / or one or more movements of one or more internal components of the surgical instrument. In at least one example, the internal component is a cutting member configured to cut tissue during a firing sequence of the surgical instrument. Additionally or alternatively, the internal component may include one or more staples configured to be fired into tissue during a firing sequence of the surgical instrument.

[0290] In at least one example, the instrument data represents one or more operations of one or more components of one or more drive assemblies (e.g., an articulation drive assembly, a closure drive assembly, a rotation drive assembly, and / or a firing drive assembly) of a surgical instrument. In at least one example, the instrument data set represents one or more operations of one or more drive members (e.g., an articulation drive member, a closure drive member, a rotation drive member, and / or a firing drive member) of a surgical instrument.

[0291] Figure 23 4 is a schematic diagram of an exemplary surgical instrument 4600 for use with process 4000, which is similar in many respects to other surgical instruments or tools described herein (e.g., surgical instrument 2112). For the sake of brevity, the present disclosure describes various aspects of process 4000 using only handheld surgical instruments. However, this is not limiting. Such aspects of process 4000 can also be implemented using robotic surgical tools (e.g., surgical tool 2117).

[0292] The surgical instrument 4600 includes a plurality of motors that can be activated to perform various functions. The plurality of motors of the surgical instrument 4600 can be activated to cause a firing motion, a closing motion, and / or an articulation in the end effector. The firing motion, the closing motion, and / or the articulation can be transmitted to the end effector of the surgical instrument 4600, for example, via a shaft assembly. However, in other examples, the surgical instrument used with the process 4000 can be configured to be able to manually perform one or more of the firing motion, the closing motion, and the articulation. In at least one example, the surgical instrument 4600 includes an end effector that treats tissue by deploying a staple into the tissue. In another example, the surgical instrument 4600 includes an end effector that treats tissue by applying therapeutic energy to the tissue.

[0293] In some cases, the surgical instrument 4600 includes a firing motor 4602. The firing motor 4602 can be operably coupled to a firing motor drive assembly 4604 that can be configured to transmit the firing motion generated by the firing motor 4602 to the end effector, specifically for moving a firing member in the form of an I-beam, which can include a cutting member, for example. In some cases, the firing motion generated by the firing motor 4602 can cause, for example, staples to be deployed from a staple cartridge into tissue captured by the end effector, and optionally, cause the cutting member of the I-beam to be advanced to cut the captured tissue.

[0294] In some cases, the surgical instrument or tool may include a closing motor 4603. The closing motor 4603 may be operably coupled to a closing motor drive assembly 4605 that is configured to transmit a closing motion generated by the closing motor 4603 to the end effector, specifically for displacing the closing tube to close the anvil and compress the tissue between the anvil and the staple cartridge. The closing motion may cause the end effector to transition from an open configuration to an approximation configuration to capture tissue, for example.

[0295] In some cases, a surgical instrument or tool may include, for example, one or more articulation motors 4606a, 4606b. The articulation motors 4606a, 4606b may be operably coupled to respective articulation motor drive assemblies 4608a, 4608b that may be configured to transmit articulation motion generated by the articulation motors 4606a, 4606b to an end effector. In some cases, the articulation motion may cause the end effector to articulate relative to an axis, for example.

[0296] In some cases, a surgical instrument or tool may include a control module 4610 that can be used with multiple motors of the surgical instrument 4600. Each of the motors 4602, 4603, 4606a, 4606b may include a torque sensor to measure the output torque on the shaft of the motor. The force on the end effector may be sensed in any conventional manner, such as by a force sensor on the outside of the jaws or by a torque sensor of the motor used to actuate the jaws.

[0297] In various cases, such as Figure 23 As shown in , the control module 4610 may include a motor driver 4626, which may include one or more H-bridge FETs. The motor driver 4626 may modulate the power delivered from a power source 4628 to a motor coupled to the control module 4610, for example, based on input from a microcontroller 4620 ("controller"). In some cases, when a motor is coupled to the control module 4610, the controller 4620 may be employed, for example, to determine the current consumed by the motor, as described above.

[0298] In some cases, the controller 4620 may include a microprocessor 4622 ("processor") and one or more non-transitory computer-readable media or memory units 4624 ("memory"). In some cases, the memory 4624 may store various program instructions that, when executed, may cause the processor 4622 to perform the various functions and / or calculations described herein. In some cases, one or more of the memory units 4624 may be coupled to the processor 4622, for example. In various cases, the processor 4622 may control the motor driver 4626 to control the position, rotational direction, and / or speed of the motor coupled to the control module 4610.

[0299] In some cases, one or more mechanisms and / or sensors (e.g., sensor 4630) may be configured to detect the force (closing force "FTC") applied by the jaws of the end effector of the surgical instrument 4600 to the tissue captured between the jaws. The FTC may be transmitted to the jaws of the end effector via the closing motor drive assembly 4605. Additionally or alternatively, the sensor 4630 may be configured to detect the force (firing force "FTF") applied to the end effector via the firing motor drive assembly 4604. In various examples, the sensor 4630 may be configured to sense closing actuation (e.g., motor current and FTC), firing actuation (e.g., motor current and FTF), articulation (e.g., angular position of the end effector), and rotation of the shaft or end effector.

[0300] One or more aspects of process 4000 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4000 may be performed by a control circuit (e.g., Figure 2A The process 4000 may be performed by a control circuit 400 comprising a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4000. Additionally or alternatively, one or more aspects of the process 4000 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2C Furthermore, process 4000 may be performed by any suitable circuitry having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0301] In various aspects, the process 4000 may be performed by the computer-implemented interactive surgical system 2100 ( Figure 19 ), the computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include the remote server 2113. The control circuitry that performs one or more aspects of process 4000 can be a component of a visualization system (e.g., visualization system 100, 160, 500, 2108) and can communicate with a surgical instrument (e.g., surgical instrument 2112, 4600) to receive instrument data therefrom. Communication between the surgical instrument and the control circuitry of the visualization system can be direct communication, or the instrument data can be routed to the visualization system via, for example, the surgical hub 2106. In at least one example, the control circuitry that performs one or more aspects of process 4000 can be a component of the surgical hub 2106.

[0302] refer to Figure 24 In various examples, visualization data 4010 is associated with instrument data 4011 by developing a composite dataset 4012 from visualization data 4010 and instrument data 4011 . Figure 24 A composite data set 4012 for the current user is shown in a graph 4013, the composite data set being developed from the current user's visualization data 4010 and the current user's instrument data 4011. Graph 4013 illustrates visualization data 4010 representing a first use cycle of surgical instrument 4600, which involves jaw positioning, clamping, and firing of surgical instrument 4600. Graph 4013 also depicts visualization data 4010 representing the beginning of a second use cycle of surgical instrument 4600, in which the jaws are repositioned for a second clamping and firing of surgical instrument 4600. Graph 4013 further depicts the current user's instrument data 4011 in the form of FTC data 4014 associated with the clamping visualization data and in the form of FTF data 4015 associated with the firing visualization data.

[0303] As described above, visualization data 4010 is derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can represent, for example, the distance between the end effector of surgical instrument 4600 and a critical structure in the surgical field of view during positioning, clamping, and / or firing of the end effector of surgical instrument 4600. In at least one example, the visualization system identifies the end effector or a component thereof in the surgical field of view, identifies the critical structure in the surgical field of view, and tracks the position of the end effector or a component thereof relative to the critical structure or relative to tissue surrounding the critical structure. In at least one example, the visualization system identifies the jaws of the end effector in the surgical field of view, identifies the critical structure in the surgical field of view, and tracks the position of the jaws relative to the critical structure or relative to tissue surrounding the critical structure during surgery.

[0304] In at least one example, the critical structure is a tumor. To remove a tumor, surgeons often prefer to cut tissue along a safety margin around the tumor to ensure that the entire tumor is removed. In such an example, visualization data 4010 can represent the distance between the jaws of the end effector and the safety margin of the tumor during positioning, clamping, and / or firing of surgical instrument 4600.

[0305] Process 4000 may further include comparing the current user's composite dataset 4012 with another composite dataset 4012', which may be received from an external source and / or derived from a previously collected composite dataset. Graph 4013 illustrates a comparison between the current user's composite dataset 4012 and another composite dataset 4012', which includes visualization data 4010' and instrument data 4011', including FTC data 4014' and FTF data 4015'. The comparison may be presented to the user of surgical instrument 4600 in real time in the form of graph 4013 or any other suitable format. The control circuitry performing one or more aspects of process 4000 may cause the comparison of the two composite datasets to be displayed on any suitable screen within the operating room (e.g., a screen of a visualization system). In at least one example, the comparison may be displayed along with real-time video of the surgical field captured on any suitable screen within the operating room. In at least one example, the control circuitry is configured to adjust instrument parameters to address detected discrepancies between the first and second composite datasets.

[0306] Additionally, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4000 may further cause the current status of instrument data (e.g., FTF data and / or FTC data) to be displayed relative to best practice equivalents. Figure 24In the example shown, the current value of FTC—represented by circle 4020—is depicted in real time against a gauge 4021 with an indicator 4022 indicating a best practice FTC. Similarly, the current value of FTF—represented by circle 4023—is depicted against a gauge 4024 with an indicator 4025 indicating a best practice FTF. Such information can be superimposed in real time on a video feed of the surgical field of view.

[0307] Figure 24 The example shown warns the user that the current FTC is above the best practice FTC and that the current FTF is also above the best practice FTF. If the current values of the FTF and / or FTC reach and / or move beyond predetermined thresholds, the control circuitry performing one or more aspects of process 4000 may further warn the current user of surgical instrument 4600 using audible, visual, and / or tactile warning mechanisms.

[0308] In some cases, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4000 may further provide projected instrument data to the current user of surgical instrument 4600 based on the current investment data. For example, Figure 24 As shown, a projected FTF 4015" is determined based on the current value of the FTF and is further displayed on the graph 4013 for the current FTF 4015 and the previously collected FTF'. Additionally or alternatively, a projected FTF circle 4026 may be displayed for the gauge 4024, as shown. Figure 24 shown.

[0309] In various aspects, the previously collected composite data set and / or best practice FTF and / or FTC is determined based on previous use of the surgical instrument 4600 in the same surgical procedure and / or other surgical procedures performed by the user, other users within the hospital, and / or users in other hospitals. For example, such data can be made available to control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that perform one or more aspects of the process 4000 by importing it from the cloud 104.

[0310] In various aspects, the control circuitry executing one or more aspects of process 4000 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can visually overlay feedback measurements of tissue thickness, compression, and stiffness on a screen displaying a real-time feed of the surgical instrument 4600 in the surgical field of view as the jaws of the end effector begin to deform the tissue captured therebetween during the clamping phase. The visual overlay correlates visual data representing tissue deformation with changes in clamping force over time. This correlation helps the user confirm correct cartridge selection, determine when to initiate firing, and determine an appropriate firing rate. This correlation can further inform the adaptive clamping algorithm. Adaptive firing rate changes can be informed by measured forces and changes in tissue motion (e.g., principal strain, tissue slip, etc.) proximate the jaws of the surgical instrument 4600, with a gauge or meter conveying the results superimposed on a screen displaying a real-time feed of the end effector in the surgical field of view.

[0311] In addition to the above, the kinematics of the surgical instrument 4600 may further indicate the operation of the instrument relative to another application or user. The kinematics may be determined via an accelerometer, a torque sensor, a force sensor, a motor encoder, or any other suitable sensor, and may generate various force, velocity, and / or acceleration data of the surgical instrument or its components for correlation with corresponding visualization data.

[0312] In various aspects, if a deviation from a best practice surgical technique is detected from the visualization data and / or instrument data, the control circuitry performing one or more aspects of process 4000 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may cause an alternative surgical technique to be presented. In at least one example, an adaptive display of instrument motion, force, tissue impedance, and results from the recommended alternative technique is presented. If the visualization data indicates that a vessel and a clip applier are detected in the surgical field of view, the control circuitry performing one or more aspects of process 4000 (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may further ensure the perpendicularity of the vessel relative to the clip applier. The control circuitry may recommend changes in position, orientation, and / or roll angle to achieve the desired perpendicularity.

[0313] In various aspects, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can perform process 4000 by comparing real-time visualization data with preoperative planning simulations. A user can simulate a surgical approach using preoperative patient scans. Preoperative planning simulations can allow a user to follow a specific preoperative plan based on training runs. The control circuitry can be configured to correlate fiducial landmarks from preoperative scans / simulations with the current visualization data. In at least one example, the control circuitry can employ object boundary tracking to establish the correlation.

[0314] When a surgical instrument interacts with tissue and deforms the surface geometry, the change in surface geometry as a function of the position of the surgical instrument can be calculated. For a given change in the position of the surgical instrument when in contact with the tissue, the corresponding change in tissue geometry can depend on the subsurface structures in the tissue region contacted by the surgical instrument. For example, in thoracic surgery, the change in tissue geometry in areas with airway substructures is different from that in areas with substantial substructures. Typically, stiffer substructures produce smaller changes in surface tissue geometry in response to a given change in the position of the surgical instrument. In various aspects, the control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to calculate a running average of the change in surgical instrument position and the change in surface geometry for a given patient, thereby providing patient-specific differences. Additionally or alternatively, the calculated running average can be compared with a second set of previously collected data. In some cases, a surface reference can be selected when no change in surface geometry is measured for each change in tool position. In at least one example, the control circuitry can be configured to determine the position of the substructure based on a change in surface geometry detected in response to a given contact between the tissue region and the surgical instrument.

[0315] In addition, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to maintain the set instrument-tissue contact throughout the tissue treatment based on a correlation between the set instrument-tissue contact and one or more tissue surface geometry changes associated with the set instrument-tissue contact. For example, the end effector of the surgical instrument 4600 can set a desired compression of the instrument-tissue contact to clamp the tissue between its jaws. The corresponding change in tissue surface geometry can be detected by the visualization system. In addition, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can derive visualization data indicative of the change in tissue surface geometry associated with the desired compression. The control circuitry can further cause the closing motor 4603 ( Figure 22) are automatically adjusted to maintain changes in tissue surface geometry associated with a desired compression. This arrangement requires continuous interaction between the surgical instrument 4600 and the visualization system to maintain changes in tissue surface geometry associated with a desired compression by continuously adjusting the compression of the jaws on the tissue based on the visualization data.

[0316] In yet another example, where the surgical instrument 4600 is a robotic tool attached to a robotic arm of a robotic surgical system (e.g., robotic system 110), the robotic surgical system may be configured to automatically adjust one or more components of the robotic surgical system to maintain contact with a set surface of tissue based on visualization data derived from changes in tissue surface geometry detected in response to contact with the set surface of tissue.

[0317] In various examples, the visualization data can be used in conjunction with measured instrument data to maintain contact between tissues in position or load control, allowing the user to manipulate the tissue to apply a predefined load to the tissue while moving the instrument relative to the tissue. The user can specify whether they want to maintain contact or maintain pressure, and visual tracking of the instrument along with the instrument's internal load can be used to enable repositioning without changing fixed parameters.

[0318] refer to Figure 25A and Figure 25B , a visualization system screen 4601 (e.g., visualization system 100, 160, 500, 2108) displays a real-time video feed of the surgical field of view during a surgical procedure. For example, the end effector 4642 of the surgical instrument 4600 includes jaws that clamp tissue near a tumor identified in the surgical field of view via a superimposed MRI image. The jaws of the end effector 4642 include an anvil 4643 and a channel that accommodates a staple cartridge. At least one of the anvil 4643 and the channel is movable relative to the other to capture tissue between the anvil 4643 and the staple cartridge. The captured tissue is then stapled via staples 4644 that can be deployed from the staple cartridge during a firing sequence of the surgical instrument 4600. Furthermore, the captured tissue is cut via a cutting member 4645 that is advanced distally during the firing sequence, but slightly behind the deployment of the staples.

[0319] like Figure 25AAs will be apparent in the figure, during the firing sequence, captured tissue and the position and / or movement of certain internal components of the end effector 4642, such as the staples 4644 and the cutting member 4645, may not be visible in the ordinary view 4640 of the real-time feed on the screen 4601. Some end effectors include windows 4641, 4653 that allow a partial view of the cutting member 4645 at the beginning and end of the firing sequence, but not during the firing sequence. As a result, the user of the surgical instrument 4600 cannot track the progress of the firing sequence on the screen 4601.

[0320] Figure 26 4030 is a logic flow diagram of a process 4030 according to at least one aspect of the present disclosure, which depicts a control program or logic configuration for synchronizing movement of a virtual representation of an end effector component with actual movement of the end effector component. The process 4030 is typically performed during a surgical procedure and includes detecting 4031 movement of an internal component of the end effector during a firing sequence, such as by overlaying 4032 a virtual representation of the internal component on the end effector, and synchronizing 4033 movement of the virtual representation on the screen 4601 with the detected movement of the internal component.

[0321] One or more aspects of process 4030 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4030 may be performed by a control circuit (e.g., Figure 2A 4030) is performed by a control circuit comprising a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4030. Additionally or alternatively, one or more aspects of process 4030 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2C Furthermore, process 4030 may be performed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described in this disclosure.

[0322] In various examples, the control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4030 may receive instrument data indicating the movement of internal components of the end effector 4642 during its firing sequence. For example, a conventional rotary encoder of the firing motor 4602 may be used to track the movement of the internal components. In other examples, the movement of the internal components may be tracked by a tracking system that employs an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” published on October 19, 2017, which is incorporated herein by reference in its entirety. In certain examples, the movement of the internal components may be tracked using one or more position sensors that may include any number of magnetic sensing elements, such as magnetic sensors categorized by whether they measure the total magnetic field or the vector components of the magnetic field.

[0323] In various aspects, process 4030 includes an overlay trigger. In at least one example, the overlay trigger can detect tissue capture by the end effector 4642. If tissue capture by the end effector 4642 is detected, process 4030 overlays a virtual representation of the cutting member 4645 in a starting position onto the end effector 4642. Process 4030 further includes projecting a staple line outlining the location where staples will be deployed into the captured tissue. Additionally, in response to the user activating a firing sequence, process 4030 causes the virtual representation of the cutting member 4645 to move distally, simulating the actual movement of the cutting member 4645 within the end effector 4642. As staples are deployed, process 4030 converts unfired staples into fired staples, allowing the user to visually track the deployment of staples and advancement of the cutting member 4645 in real time.

[0324] In various examples, for example, a control circuit (e.g., control circuit 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4030 may be instructed to close motor 4603 ( Figure 22) The control circuitry can detect tissue capture by the end effector 4642 based on instrument data indicating the force applied to the jaws of the end effector 4642 by closing the motor drive assembly 4605. The control circuitry can further determine the position of the end effector within the surgical field of view based on visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108). In at least one example, the end effector position can be determined relative to a reference point (e.g., a critical structure) in the tissue.

[0325] In any case, the control circuit causes the virtual representation of the internal components to be superimposed on the end effector 4642 on the screen 4601 at a position commensurate with the position of the internal components within the end effector. The control circuit further causes the projected virtual representation of the internal components to move synchronously with the internal components during the firing sequence. In at least one example, synchronization is improved by incorporating markings on the end effector 4642, which the control circuit can use as reference points to determine where to superimpose the virtual representation.

[0326] Figure 25B An augmented view 4651 of a live feed of the surgical field on screen 4601 is shown. Figure 25B In the enhanced view 4651 of the example, a virtual representation of staples 4644 and cutting members 4645 are superimposed on the end effector 4642 during the firing sequence. The superposition distinguishes fired staples 4644a from unfired staples 4644b and the completed cut line 4646a from the projected cut line 4646b that tracks the progress of the firing sequence. The superposition further shows the starting point of the staple line 4647 and the projected end 4649 of the staple line that will not reach the end of the tissue in the cut line. In addition, based on the superposition of the tumor MRI image, a safety margin distance "d" between the tumor and the projected cut line 4646b is measured and presented with the superposition. The superimposed safety margin distance "d" assures the user that all of the tumor will be removed.

[0327] like Figure 25B As shown, the control circuit is configured to enable the visualization system to continuously reposition the virtual representation of the internal component to correlate with the actual movement of the internal component. Figure 25B In the example of FIG, the overlay shows that the completed cut line 4646 lags slightly behind the fired staple line 4644a by a distance “d1”, which assures the user that the firing sequence is proceeding correctly.

[0328] Now refer to Figure 27, a visualization system (e.g., visualization system 100, 160, 500, 2108) can detect and / or define the trocar position using tool lighting 4058 and camera 4059. Based on the determined trocar position, the user can be guided to the most appropriate trocar port to accomplish the intended function based on time efficiency, location of critical structures, and / or avoidance or risk.

[0329] Figure 27 Three trocar positions (Trocar 1, Trocar 2, Trocar 3) are shown extending through the body wall 4050 at different positions and orientations relative to the body wall and relative to a critical structure 4051 in a cavity 4052 within the body wall 4050. The trocars are represented by arrows 4054, 4055, 4056. An illumination tool 4058 can be used to detect the trocar positions by using cascaded light or an image of the surrounding environment. Additionally, the light source of the illumination tool 4058 can be a rotatable light source. In at least one example, the light source of the illumination tool 4058 and the camera 4059 are used to detect the distance of the trocars relative to a target location (e.g., critical structure 4051). In various examples, if a more preferred instrument position is determined based on visualization data obtained by the camera 4059 recording the light projected by the light source of the illumination tool 4058, the visualization system can suggest a change in the position of the instrument. Screen 4060 may display the distance between the cannula and the target tissue, whether tool entry through the cannula is acceptable, the risks associated with utilizing the cannula, and / or the expected operating time using the cannula, which may help the user select the optimal cannula for introducing the surgical tool into cavity 4052.

[0330] In various aspects, a surgical hub (e.g., surgical hub 2106, 2122) can recommend an optimal trocar for inserting a surgical tool into cavity 4052 based on user characteristics, such as those that can be received from a user database. User characteristics include user hand dominance, patient-side user preferences, and / or user physical characteristics (e.g., height, arm length, range of motion). The surgical hub can utilize these characteristics, position and orientation data of available trocars, and / or position data of key structures to select the optimal trocar for inserting the surgical tool in an effort to reduce user fatigue and increase efficiency. In various aspects, if the user inverts the end effector orientation, the surgical hub can further invert the control of the surgical instrument.

[0331] The surgical hub can reconfigure the output of the surgical instrument based on the visualization data. For example, if the visualization data indicates that the surgical instrument is being retracted or is being used to perform a different task, the surgical hub can prohibit activation of the surgical instrument's therapeutic energy output.

[0332] In various aspects, the visualization system can be configured to track the blood surface or estimate blood volume based on reflected IR or red light wavelengths to depict blood based on non-blood surface and surface geometry measurements. This can be reported as an absolute static measurement or a rate of change to provide quantitative data on the amount and degree of change of bleeding.

[0333] refer to Figure 28 , various elements (e.g., structured light projector 706 and camera 720) of a visualization system (e.g., visualization system 100, 160, 500, 2108) can be used to generate visualization data of the anatomical organ, thereby generating a virtual 3D structure 4130 of the anatomical organ.

[0334] As described herein, structured light in the form of stripes or lines, for example, can be projected from a light source and / or projector 706 onto a surface 705 of a target anatomical structure to identify the shape and contours of the surface 705. Figure 1 ) can be configured to detect a projected pattern of light on surface 705. The way in which the projected pattern deforms when it strikes surface 705 allows the vision system to calculate depth and surface information of the target anatomy.

[0335] Figure 29 A logic flow diagram depicting a process 4100 of a control program or logic configuration according to at least one aspect of the present disclosure is provided. In various instances, the process 4100 identifies 4101 a surgical procedure and identifies 4102 an anatomical organ targeted by the surgical procedure. The process 4100 further generates 4104 a virtual 3D configuration 4130 of at least a portion of the anatomical organ, identifies 4105 an anatomical structure of at least a portion of the anatomical organ associated with the surgical procedure, couples 4106 the anatomical structure to the virtual 3D configuration 4130, and overlays 4107 a layout plan for the surgical procedure, determined based on the anatomical structure, onto the virtual 3D configuration 4130.

[0336] One or more aspects of process 4100 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4100 may be performed by a control circuit (e.g., Figure 2A The process 4100 may be performed by a control circuit 400 comprising a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of the process 4100. Additionally or alternatively, one or more aspects of the process 4100 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2CFurthermore, one or more aspects of process 4100 may be performed by any suitable circuitry having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0337] In various aspects, the process 4100 may be performed by the computer-implemented interactive surgical system 2100 ( Figure 19 ), the computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include a remote server 2113. The control circuitry that performs one or more aspects of the process 4100 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0338] Control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4100 may identify 4101 a surgical procedure and / or identify 4102 an anatomical organ targeted by the surgical procedure by retrieving such information from a database storing such information or obtaining the information directly from user input. In at least one example, the database is stored in a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). In at least one example, the database includes a hospital EMR.

[0339] In one aspect, the surgical system 2200 includes a surgical hub 2236 connected to multiple operating room devices, such as a visualization system located in the operating room (e.g., visualization systems 100, 160, 500, 2108). In at least one example, the surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the visualization system, the cloud 2204, and / or the remote server 2213. Control circuitry of the surgical hub 2236 that performs one or more aspects of the process 4100 can identify 4101 a surgical procedure and / or identify 4102 an anatomical organ targeted by the surgical procedure by retrieving such information from a database stored in the cloud 2204 and / or the remote server 2213.

[0340] Control circuitry performing one or more aspects of process 4100 may cause a visualization system (e.g., visualization system 100, 160, 500, 2108) to perform an initial scan of at least a portion of the anatomy to generate 4104 a three-dimensional ("3D") configuration 4130 of at least a portion of the anatomy targeted for surgical intervention. Figure 28 In the example shown, the anatomical organ is a stomach 4110. The control circuitry can cause one or more components of the visualization system (e.g., structured light projector 706 and camera 720 utilizing structured light 4111) to generate visualization data by performing a scan of at least a portion of the anatomical organ while the camera is introduced into the body. The 3D configuration of at least a portion of the anatomical organ can be generated using current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input.

[0341] In addition, the control circuitry performing one or more aspects of process 4100 identifies 4105 an anatomical structure of at least a portion of an anatomical organ that is relevant to the surgical procedure. In at least one example, a user can select the anatomical structure using any suitable input device. Additionally or alternatively, the visualization system can include one or more imaging devices 120 having spectral cameras (e.g., hyperspectral cameras, multispectral cameras, or selective spectral cameras) configured to detect reflected spectral waveforms and generate images based on molecular responses to different wavelengths. The control circuitry can utilize light absorption or refraction properties of tissue to distinguish between different tissue types of the anatomical organ, thereby identifying relevant anatomical structures. Furthermore, the control circuitry can utilize current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), stored visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input to identify relevant anatomical structures.

[0342] The identified anatomical structure can be an anatomical structure in the surgical field of view and / or the anatomical structure can be selected by the user. In various examples, the position tracking of the relevant anatomical structure can be extended beyond the current visible view of the camera pointed at the surgical field of view. In one example, this is achieved by using common visible coupled landmarks or by using secondary coupled motion tracking. For example, secondary tracking can be accomplished using a secondary imaging source, calculation of range motion, and / or using pre-established beacons measured by a second visualization system.

[0343] As above combined Figure 14As described in more detail, the visualization system may utilize structured light projector 706 to project an array of patterns or lines, wherein camera 720 may determine the distance to the target location. The visualization system may then transmit the pattern or lines of known dimensions at a set distance equal to the determined distance. Furthermore, the spectral camera may determine the dimensions of the pattern, which may vary based on the light absorption or refraction properties of the tissue at the target location. The difference between the known dimensions and the determined dimensions indicates the tissue density at the target location, which is indicative of the tissue type at the target location. Control circuitry performing one or more aspects of process 4100 may identify relevant anatomical structures based at least in part on the determined tissue density at the target location.

[0344] In at least one example, the detected abnormal tissue density may be associated with a disease state. In addition, the control circuit selects, updates or modifies one or more settings of a surgical instrument for treating tissue based on the tissue density detected via the visualization data. For example, the control circuit may change various clamping and / or firing parameters of a surgical stapler used to suturing and cutting tissue. In at least one example, the control circuit may slow down the firing sequence and / or allow more clamping time based on the tissue density detected by the visualization data. In various examples, the control circuit may warn the user of the surgical instrument of abnormal tissue density by, for example, displaying on a screen instructions to reduce the bite size, increase or decrease the energy delivery output of the electrosurgical instrument, and adjust the amount of jaw closure. In another example, if the visualization data indicates that the tissue is fat tissue, the instruction may be to increase the power to reduce the energy application time.

[0345] Furthermore, identifying the type of surgical procedure can aid the control circuitry in identifying the target organ. For example, if the procedure is a left upper lobectomy, the lung is likely the target organ. Therefore, the control circuitry will only consider visual and non-visual data related to the lung and / or tools typically used in this type of surgery. Furthermore, knowledge of the surgical procedure type can better inform other image fusion algorithms, such as tumor location and staple line placement.

[0346] In various aspects, knowledge of the operating table position and / or insufflation pressure can be used by control circuitry performing one or more aspects of process 4100 to establish a baseline position of the target anatomical organ and / or related anatomical structures identified from visualization data. Movement of the operating table (e.g., moving the patient from a flat position to a reverse Trendelenburg position) can result in deformation of the anatomical structure, which can be tracked and compared to the baseline to continuously inform the position and status of the target organ and / or related anatomical structure. Similarly, changes in insufflation pressure within a body cavity can perturb the baseline visualization data of the target organ and / or related anatomical structure within the body cavity.

[0347] Control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may perform one or more aspects of a process that derives baseline visualization data of a patient's target organs and / or related anatomical structures on an operating table during a surgical procedure, determines a change in the position of the operating table, and re-derives baseline visualization data of the patient's target organs and / or related anatomical structures in the new position.

[0348] Likewise, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) may perform one or more aspects of a process for deriving baseline visualization data of a target organ and / or related anatomical structure of a patient on an operating table during a surgical procedure, determining a change in insufflation pressure in a body cavity of the patient, and re-deriving baseline visualization data of the target organ and / or related anatomical structure of the patient at the new insufflation pressure.

[0349] In various cases, the control circuitry performing one or more aspects of process 4100 may couple the identified anatomical structure to the virtual 3D construct by superimposing landmarks or markers onto the virtual 3D construct of the organ to indicate the location of the anatomical structure, such as Figure 28 As shown. The control circuitry can also cause user-defined structures and tissue planes to be superimposed on the virtual 3D construct. In various aspects, a hierarchy of tissue types can be established to organize the anatomical structures identified on the virtual 3D construct. Table 1 provided below lists an exemplary hierarchy for the lungs and stomach.

[0350] organ Layer 1 Layer 2 Layer 3 lung Left lung Left upper lobe Segments of the left upper lobe, major blood vessels / airways Stomach Stomach Gastric fundus, antrum, and pylorus His angle, notch angle, greater bend / lesser bend

[0351] In various aspects, relevant anatomical structures identified on the virtual 3D construct can be renamed and / or repositioned by the user to correct errors or according to preferences. In at least one example, corrections can be voice-activated. In at least one example, corrections are recorded for future machine learning.

[0352] In addition to the above, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) executing one or more aspects of process 4100 may overlay 4107 a surgical layout plan (e.g., layout plan 4120) onto a virtual 3D configuration of the target organ (e.g., stomach 4110). In at least one example, the virtual 3D configuration is displayed on a separate screen of the visualization system from the screen displaying the live feed / view of the surgical field of view. In another example, a single screen may alternately display the live feed of the surgical field of view and the 3D configuration. In such examples, a user may alternate between the two views using any suitable input device.

[0353] exist Figure 28In the example shown, the control circuitry has determined that the surgical procedure is a sleeve gastrectomy and that the target organ is the stomach. In an initial scan of the abdominal cavity, the control circuitry uses visualization data (e.g., structured light data and / or spectral data) to identify the stomach, liver, spleen, greater curvature of the stomach, and pylorus. This is informed by knowledge of the procedure and the structures of interest.

[0354] Visualization data (e.g., structured light data and / or spectral data) can be utilized by control circuitry to identify stomach 4110, liver, and / or spleen by comparing current structured light data with stored structured light data previously associated with such organs. In at least one example, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) can utilize structured light data representing characteristic anatomical contours of the organs and / or spectral data representing characteristic subsurface tissue features to identify anatomical structures associated with a surgical layout plan (e.g., layout plan 4120).

[0355] In at least one example, the visualization data can be used to identify the pyloric vein 4112 indicating the location 4113 of the pylorus 4131, identify the gastroepiploic vessels 4114 indicating the location 4115 of the greater curvature of the stomach 4110, identify the bend 4116 in the right gastric vein indicating the location 4117 of the notch angle 4132, and / or identify the location 4119 of the angle of His 4121. The control circuit can assign a landmark to one or more of the identified locations. In at least one example, as Figure 28 As shown, the control circuitry causes the visualization system to overlay landmarks at locations 4113, 4117, and 4119 on a virtual 3D construct of the stomach 4110 generated using the visualization data, as described above. In various aspects, the landmarks can be overlaid simultaneously on the virtual 3D construct and the surgical field view, allowing the user to switch between views without losing sight of the landmarks. The user can zoom out of the view on the screen displaying the virtual 3D construct to display the overall layout plan, or zoom in to display a portion similar to the surgical field view. The control circuitry can continuously track and update the landmarks.

[0356] During a sleeve gastrectomy, the surgeon typically sutures the gastric tissue 4 cm or approximately 4 cm from the pylorus. Prior to suturing, at the start of the sleeve gastrectomy, an energy device is introduced into the patient's abdominal cavity to dissect the gastroepiploic artery and omentum distal to the greater curvature and 4 cm or approximately 4 cm from the pylorus. As described above, the control circuit, having identified position 4113, can automatically superimpose the end effector of the energy device 4 cm or approximately 4 cm from position 4113. Superimposition of the end effector of the energy device or any suitable landmark 4 cm or approximately 4 cm from the pylorus identifies the starting position of the sleeve gastrectomy.

[0357] As the surgeon dissects along the greater curvature of the stomach, the control circuit causes the landmark at position 4113 and / or the superimposed end effector of the energy device to be removed. As the surgeon approaches the spleen, a distance indicator is automatically superimposed on the virtual 3D anatomy view and / or the surgical field of view. The control circuit can cause the distance indicator to identify a distance of 2 cm from the spleen. For example, when the anatomical path reaches or is about to reach 2 cm from the spleen, the control circuit can cause the distance indicator to flash and / or change color. The distance indicator superposition remains until the user reaches position 4119 at the His angle 4121.

[0358] refer to Figure 30 Once the surgical stapler is introduced into the abdominal cavity, the control circuit can use the visual data to identify the pylorus 4131, the notch angle 4132, the greater curvature 4133 of the stomach 4110, the lesser curvature 4134 of the stomach 4110 and / or other anatomical structures related to sleeve gastrectomy. The overlay of the bougie can also be displayed. The introduction of the surgical instrument into the body cavity (for example, the introduction of the surgical stapler into the abdominal cavity) can be detected by the control circuit based on the visual data indicating the visual prompts on the end effector (such as a unique color, mark and / or shape). The control circuit can identify the surgical instrument in a database that stores such visual prompts and corresponding visual prompts. Alternatively, the control circuit can prompt the user to identify the surgical instrument inserted into the body cavity. Alternatively, a surgical trocar that facilitates entry into the body cavity may include one or more sensors for detecting the surgical instrument inserted therethrough. In at least one example, the sensor includes an RFID reader that is configured to identify the surgical instrument from an RFID chip on the surgical instrument.

[0359] In addition to identifying landmarks of relevant anatomical structures, the visualization system may also overlay a surgical layout plan 4135, which may be in the form of a suggested treatment path, onto the 3D critical structures and / or surgical field view. Figure 30 In the example of , the surgical procedure is a sleeve gastrectomy, and the surgical layout plan 4135 is in the form of three resection paths 4136, 4137, 4138 and the corresponding resulting volumes of the resulting sleeve.

[0360] like Figure 30 As shown, the distances (a, a1, a2) from the pylorus 4131 to the starting point for forming the gastric sleeve are shown. Each starting point produces a different gastric sleeve size (e.g., 400cc, 425cc, and 450cc, respectively, for starting points 4146, 4147, and 4148 at distances a, a1, and a2 from the pylorus 4131). In one example, the control circuit prompts the user to enter a size selection and, in response, presents a surgical layout plan, which may be in the form of a resection path, that produces the selected gastric sleeve size. In another example, as shown, Figure 30As shown, the control circuit presents a plurality of resection paths 4136, 4137, 4138 and corresponding gastric sleeve sizes. The user may then select one of the recommended resection paths 4136, 4137, 4138, and in response, the control circuit removes the unselected resection paths.

[0361] In yet another example, the control circuitry allows the user to adjust the recommended resection path on a screen that displays the resection path superimposed on a virtual 3D configuration and / or surgical field of view. The control circuitry can calculate the size of the gastric sleeve based on these adjustments. Alternatively, in another example, the user can select a starting point to form the gastric sleeve at a desired distance from the pylorus 4131. In response, the control circuitry calculates the size of the gastric sleeve based on the selected starting point.

[0362] For example, presenting the resection path can be achieved by causing the visualization system to overlay the resection path onto the virtual 3D anatomy view and / or the surgical field view. Conversely, removing the recommended resection path can be achieved by causing the visualization system to remove the overlay of the resection path from the virtual 3D anatomy view and / or the surgical field view.

[0363] Still refer to Figure 30 In some examples, once the end effector of the surgical stapler clamps stomach tissue between a starting point selected from recommended starting points 4146, 4147, 4147 and an end position 4140 at a predefined distance from the notch angle 4132, the control circuit presents information regarding clamping and / or firing the surgical stapler. In at least one example, as Figure 24 As shown, a composite data set 4012 from visualization data 4010 and instrument data 4011 can be displayed. Additionally or alternatively, FTC values and / or FTF values can be displayed. For example, the current value of FTC—represented by circle 4020—can be depicted in real time against a gauge 4021 with an indicator 4022 indicating a best practice FTC. Similarly, the current value of FTF—represented by circle 4023—can be depicted against a gauge 4024 with an indicator 4025 indicating a best practice FTF.

[0364] After the surgical stapler is fired, the suggestion for the new cartridge selection can be presented to the screen of the surgical stapler or any screen of the visualization system, as described in more detail below. When the surgical stapler is removed from the abdominal cavity, the selected staple cartridge is reloaded and reintroduced into the abdominal cavity, a distance indicator - identifying a constant distance (d) from multiple points along the lesser curvature 4134 of the stomach 4110 to the selected resection path - is superimposed on the virtual 3D construction view and / or the surgical field view. In order to ensure the correct orientation of the end effector of the surgical stapler, the distance from the target to the distal end of the end effector of the surgical stapler and the distance from the proximal end to the previously fired staple line are superimposed on the virtual 3D construction view and / or the surgical field view. Repeat this process until the resection is complete.

[0365] The one or more distances recommended and / or calculated by the control circuitry can be determined based on stored data. In at least one example, the stored data includes preoperative data, user preference data, and / or data from surgical procedures previously performed by the user or other users.

[0366] refer to Figure 31 In accordance with at least one aspect of the present disclosure, process 4150 depicts a control program or logic configuration for recommending a resection path for removing a portion of an anatomical organ. Process 4150 identifies 4151 an anatomical organ targeted for surgical intervention, identifies 4152 anatomical structures of the anatomical organ associated with the surgical intervention, and recommends 4153 a surgical resection path for removing the portion of the anatomical organ using a surgical instrument, as described in greater detail elsewhere herein in conjunction with process 4100. The surgical resection path is determined based on the anatomical structure. In at least one example, the surgical resection path includes different starting points.

[0367] One or more aspects of process 4150 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4150 may be performed by a control circuit (e.g., Figure 2A 4150) is performed by a control circuit 400 that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4150. Additionally or alternatively, one or more aspects of process 4150 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2CFurthermore, process 4150 may be performed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0368] refer to Figures 32A to 32D , control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4100 or process 4150 can utilize dynamic visualization data to update or modify a surgical layout plan in real time during implementation. In at least one example, the control circuitry updates or modifies a planned resection path (e.g., a surgical instrument) for removing a portion of an organ or an abnormality (e.g., a tumor or region) based on dynamic visualization data from one or more imaging devices of a visualization system (e.g., visualization system 100, 160, 500, 2108). Figure 32B ) is modified to an alternative resection path ( Figure 32D ), the visualization system tracks the progress of the resected tissue and surrounding tissue. Resection path modifications can be triggered by the position of a critical structure (e.g., a blood vessel) moving into the resection path. For example, the tissue resection process sometimes results in tissue inflammation that changes the shape and / or volume of the tissue, which can cause critical structures (e.g., blood vessels) to shift. The dynamic visualization data enables the control circuit to detect changes in the position and / or volume of the critical structure and / or related anatomical structures near the set resection path. If the change in position and / or volume causes the critical structure to move into the resection path or within the safety margin of the resection path, the control circuit modifies the set resection path by selecting or at least suggesting an alternative resection path for the surgical instrument.

[0369] Figure 32A A real-time view 4201 of a surgical field on a screen 4230 of a visualization system is shown. A surgical instrument 4200 is introduced into the surgical field to remove a target area 4203. An initial planned layout 4209 for removing the area is superimposed on the real-time view 4201, as shown in FIG. Figure 32B 4203 is surrounded by key structures 4205, 4206, 4207, and 4208. Figure 32B As shown, the initial plan layout 4209 extends a resection path from region 4203 around region 4203 with a predefined safety margin. The resection path avoids crossing or penetrating critical structures by extending on the outside (e.g., critical structure 4208) or on the inside (e.g., critical structure 4206) of the critical structure. As described above, the initial plan layout 4209 is determined by the control circuit based on visualization data from the visualization system.

[0370] Figure 32CA real-time view 4201' of the surgical field of view on the screen 4230 of the visualization system at a later time (00:43) is shown. The end effector 4202 of the surgical instrument 4200 resects tissue along a predefined resection path defined by the layout plan 4209. For example, a volumetric change in tissue including the region 4203 due to tissue inflammation causes the critical structures 4206 and 4208 to shift into the predefined resection path. In response, the control circuit recommends an alternative resection path 4210 that bypasses the critical structures 4206 and 4208, which protects the critical structures 4206 and 4208 from damage, as shown in FIG. Figure 32D In various examples, alternative resection paths can be recommended to optimize the amount of healthy tissue that will be preserved, and guidance can be provided to the user to ensure they avoid critical structures, which will minimize bleeding and therefore reduce surgical time and stress of dealing with unexpected situations, while balancing the impact on the remaining organ volume.

[0371] In various aspects, control circuitry executing one or more aspects of one or more processes described herein can receive and / or export visualization data from multiple imaging devices of a visualization system. The visualization data can facilitate tracking of critical structures outside the real-time view of the surgical field of view. Common landmarks can allow the control circuitry to merge visualization data from multiple imaging devices of the visualization system. In at least one example, secondary tracking of critical structures outside the real-time view of the surgical field of view can be achieved, for example, via a secondary imaging source, calculation of range motion, or via pre-established beacons / landmarks measured by a second system.

[0372] Usually refer to Figures 33 to 35 In accordance with at least one aspect of the present disclosure, a logic flow diagram of process 4300 depicts a control program or logic configuration for presenting or superimposing parameters of a surgical instrument on or near a recommended surgical resection path. Process 4300 is typically performed during a surgical procedure and includes identifying 4301 an anatomical organ targeted for surgical procedure, identifying 4302 anatomical structures associated with the surgical procedure based on visualization data from at least one imaging device, and recommending 4303 a surgical resection path for removing a portion of the anatomical organ using a surgical instrument. In at least one example, the surgical resection path is determined based on the anatomical structure. Process 4300 further includes presenting 4304 the parameters of the surgical instrument based on the surgical resection path. Additionally or alternatively, process 4300 further includes adjusting 4305 the parameters of the surgical instrument based on the surgical resection path.

[0373] One or more aspects of process 4300 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4300 may be performed by a control circuit (e.g., Figure 2A 4030) is performed by a control circuit comprising a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4030. Additionally or alternatively, one or more aspects of process 4300 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2C Furthermore, process 4300 may be performed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0374] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4300 may identify 4301 an anatomical organ targeted for surgical procedure, identify 4302 anatomical structures associated with the surgical procedure based on visualization data from at least one imaging device of a visualization system (e.g., visualization system 100, 160, 500, 2108), and / or recommend 4303 a surgical resection path for removing a portion of the anatomical organ with a surgical instrument (e.g., surgical instrument 4600), as described elsewhere herein in conjunction with process 4150 ( Figure 31 )、4100( Figure 29 ) as described. In addition, the control circuitry performing one or more aspects of process 4300 may recommend or suggest one or more parameters for a surgical instrument based on the recommended 4303 surgical resection path. In at least one example, the control circuitry presents the suggested parameters 4304 for the surgical instrument by superimposing such parameters on or near the recommended surgical path, such as Figure 34 and Figure 35 shown.

[0375] Figure 34 A virtual 3D configuration 4130 of a patient's stomach undergoing sleeve gastrectomy using surgical instrument 4600 is shown in accordance with at least one aspect of the present disclosure. Figure 28In more detail, for example, various elements (e.g., structured light projector 706 and camera 720) of a visualization system (e.g., visualization systems 100, 160, 500, 2108) can be used to generate visualization data to form a virtual 3D construct 4130. Based on the visualization data from one or more imaging devices of the visualization system, relevant anatomical structures (e.g., pylorus 4131, notch angle 4132, His angle 4121) are identified. In at least one example, landmarks are assigned to the locations 4113, 4117, 4119 of such anatomical structures by overlaying the landmarks onto the virtual 3D construct 4130.

[0376] Additionally, based on the identified anatomical structure, a surgical resection path 4312 is recommended 4303. In at least one example, the control circuitry overlays the surgical resection path 4312 onto the virtual 3D construct 4130, as shown in FIG. Figure 34 As described in more detail elsewhere herein, the recommended surgical path can be automatically adjusted based on the desired volume output. Additionally, the projection margins can be automatically adjusted based on key structures and / or tissue anomalies automatically identified by the control circuitry based on the visualization data.

[0377] In various aspects, the control circuitry performing at least one aspect of process 4300 presents parameters 4314 for the surgical instrument selected based on the recommended 4303 surgical resection path 4312. Figure 34 In the example shown, parameters 4314 indicate automatic selection of a staple cartridge for use with surgical instrument 4600 when performing a sleeve gastrectomy based on the recommended 4303 surgical resection path. In at least one example, parameters 4314 include at least one of staple cartridge size, staple cartridge color, staple cartridge type, and staple cartridge length. In at least one example, the control circuitry presents 4304 parameters 4314 of surgical instrument 4600 by superimposing such parameters on or near the recommended 4303 surgical resection path 4312, as shown in FIG. Figure 34 and Figure 35 shown.

[0378] In various aspects, control circuitry performing at least one aspect of process 4300 presents tissue parameters 4315 along one or more portions of surgical resection path 4312. Figure 34 In the example shown, tissue parameter 4315 is presented as tissue thickness by displaying a cross-section taken along line AA, which represents the thickness of tissue along at least a portion of surgical resection path 4312. In various aspects, the staple cartridge utilized by surgical instrument 4600 can be selected based on tissue parameter 4315. For example, Figure 34As shown, a black staple cartridge including a larger staple size is selected for use in the thicker antral muscle tissue, while a green staple cartridge including a smaller staple size is selected for use in the myocardial tissue of the gastric body and fundus portions.

[0379] The tissue parameters 4315 include at least one of tissue thickness, tissue type, and volume outcome of the sleeve gastrectomy resulting from the recommended surgical resection path 4312. The tissue parameters 4315 may be derived from previously captured CT, ultrasound, and / or MRI images of the patient's organ and / or from a previously known average tissue thickness. In at least one example, the surgical instrument 4600 is a smart instrument (similar to the smart instrument 2112), and the tissue thickness and / or selected staple cartridge information is transmitted to the surgical instrument 4600 for optimizing the closure setting, the firing setting, and / or any other suitable surgical instrument setting. In one example, the tissue thickness and / or selected staple cartridge information may be transmitted to the surgical instrument 4600 from a surgical hub (e.g., surgical hub 2106, 2122) that communicates with a visualization system (e.g., visualization system 100, 160, 500, 2108) and the surgical instrument 4600, as in combination with a surgical hub. Figures 17 to 19 described.

[0380] In various examples, the control circuitry performing at least one aspect of process 4300 recommends an arrangement 4317 of two or more staple cartridge sizes (e.g., 45 mm and 60 mm) based on a tissue thickness determined along at least a portion of surgical resection path 4312. Figure 35 As shown, the control circuitry can further present an arrangement 4317 along the surgical resection path 4312 recommended 4303. Alternatively, the control circuitry can present a suitable arrangement 4317 along the surgical resection path selected by the user. As described above, the control circuitry can determine the tissue thickness along the user-selected resection path and recommend a staple cartridge arrangement based on the tissue thickness.

[0381] In various aspects, the control circuitry performing one or more aspects of process 4300 can recommend a surgical resection path, or optimize the selected surgical resection path, to minimize the number of staple cartridges in staple cartridge arrangement 4317 without compromising the size of the resulting gastric sleeve beyond a predetermined threshold. Reducing the number of exhausted cartridges reduces surgical time and cost, and reduces trauma to the patient.

[0382] Still refer to Figure 35, arrangement 4317 includes a first staple cartridge 4352 and a last staple cartridge 4353 that define the starting and ending points of the surgical resection path 4312. In the event that only a fraction of the last staple cartridge 4353 of the recommended staple cartridge arrangement 4317 is required, the control circuitry can adjust the surgical resection path 4312 to eliminate the need for the last staple cartridge 4353 without compromising the size of the resulting gastric sleeve beyond a predetermined threshold.

[0383] In various examples, a control circuit performing at least one aspect of process 4300 presents a virtual firing of a recommended staple cartridge arrangement 4317 that virtually divides the virtual 3D structure 4130 into a retained portion 4318 and a removed portion 4319, as shown in FIG. Figure 35 As shown. The retained portion 4318 is a virtual representation of the gastric sleeve that is created by firing the staple cartridge arrangement 4317 to implement the recommended surgical resection path 4312. The control circuitry can further determine an estimated volume of the retained portion 4318 and / or the removed portion 4319. The volume of the retained portion 4318 represents the volume of the resulting gastric sleeve. In at least one example, the volume of the retained portion 4318 and / or the removed portion 4319 is derived from visualization data. In another example, the volume of the retained portion 4318 and / or the removed portion 4319 is determined by a database that stores retained volumes, removed portion volumes, and corresponding surgical resection paths. The database can be constructed from previous surgical procedures performed on organs of the same or at least similar size that have been resected using the same or at least similar resection paths.

[0384] In various examples, a combination of predetermined average tissue thickness data based on situational awareness of the organ, as described in greater detail above, combined with volumetric analysis from a visualization source and secondary imaging such as CT, MRI, and / or ultrasound, if available for the patient, can be used to select the first staple cartridge for arrangement 4317. In addition to the visualization data, the firing of subsequent staple cartridges in arrangement 4317 can utilize instrument data from previous firings to optimize firing parameters. For example, instrument data that can be used to supplement volumetric measurements include FTF, FTC, current draw of the motor driving the firing and / or closure, end effector closure gap, firing rate, tissue impedance measurements at the jaws, and / or wait or pause times during use of the surgical instrument.

[0385] In various examples, visualization data (e.g., structured light data) can be used to track changes in surface geometry in tissue being treated by a surgical instrument (e.g., surgical instrument 4600). Additionally, visualization data (e.g., spectral data) can be used to track key structures beneath the tissue surface. Structured data and / or spectral data can be used to maintain a defined instrument-tissue contact throughout tissue treatment.

[0386] In at least one example, the end effector 4642 of the surgical instrument 4600 can be used to grasp tissue between its jaws. Once desired tissue-instrument contact is confirmed by user input, for example, visualization data of the end effector and surrounding tissue associated with the desired tissue-instrument contact can be used to automatically maintain the desired tissue-surface contact during at least a portion of the tissue treatment. The desired tissue-surface contact can be automatically maintained by, for example, slight manipulation of the position, orientation, and / or FTC parameters of the end effector 4642.

[0387] In the case where the surgical instrument 4600 is a handheld surgical instrument, the display 4625 ( Figure 22 ) in the form of instructions on the surgical instrument 4600. The surgical instrument 4600 may also issue an alarm when user manipulation is required to re-establish the desired tissue-surface contact. At the same time, for example, non-user manipulations (e.g., manipulations of FTC parameters and / or articulation angles) may be transmitted from the surgical hub 2106 or the visualization system 2108 to the controller 4620 of the surgical instrument 4600. The controller 4620 may then cause the motor driver 4626 to implement the desired manipulation. In the case where the surgical instrument 4600 is a surgical tool coupled to a robotic arm of a robotic system 2110, for example, the position and / or orientation manipulations may be transmitted from the surgical hub 2106 or the visualization system 2108 to the robotic system 2110.

[0388] Main references Figures 36A to 36C , shows the firing of the surgical instrument 4600 loaded with the first staple cartridge 4652 in the staple cartridge arrangement 4317. In the first stage, as Figure 36A As shown, a first landmark 4361 and a second landmark 4362 are superimposed on the surgical resection path 4312. The landmarks 4361 and 4362 are separated by a distance (d1) defined by the size of the staple cartridge 4652 (e.g., 45), which represents the length of the staple line 4363 to be deployed by the staple cartridge 4652 onto the surgical resection path 4312. The control circuitry performing one or more aspects of the process 4300 can employ visualization data, as described in greater detail elsewhere herein, to superimpose the landmarks 4361 and 4362 onto the surgical resection path 4312 and continuously track and update their positions relative to predetermined key structures (e.g., anatomical structures 4364, 4365, 4366, 4367).

[0389] During firing, if Figure 36BAs shown, the staples of staple line 4363 are deployed into the tissue, and cutting member 4645 is advanced to cut the tissue along surgical resection path 4312 between landmarks 4361, 4362. In various instances, advancement of cutting member 4645 causes stretching and / or displacement of the treated tissue. Tissue stretching and / or displacement exceeding a predetermined threshold indicates that cutting member 4645 is moving too quickly through the treated tissue.

[0390] Figure 37 41 is a logic flow diagram of a process 4170 that depicts a control program or logic configuration for adjusting the firing velocity of a surgical instrument to account for tissue stretch / displacement during firing. The process 4170 includes monitoring 4171 tissue stretch / displacement during firing of the surgical instrument and adjusting 4173 firing parameters if 4172 tissue stretch / displacement is greater than or equal to a predetermined threshold.

[0391] One or more aspects of process 4170 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4170 may be performed by a control circuit (e.g., Figure 2A 4170) is performed by a control circuit 400 that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4170. Additionally or alternatively, one or more aspects of process 4170 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2C Furthermore, process 4170 may be performed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0392] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4170 monitors 4171 tissue stretch / displacement during firing of surgical instrument 4600 using visualization data from a visualization system (e.g., visualization system 100, 160, 500, 2108). Figure 36BIn the example shown, tissue stretch / displacement (d) is monitored 4171 by using visualization data to track distortion in the structured light grid projected onto the tissue during firing and / or tracking landmarks 4364, 4365, 4366, 4367 representing the locations of adjacent anatomical structures. Additionally or alternatively, tissue stretch (d) can be monitored 4171 by tracking the location of landmark 4362 during firing. Figure 36B In the example of , tissue stretch / displacement (d) is the difference between the distance (d1) between the landmarks 4361, 4362 during firing and the distance (d2) between the landmarks 4361, 4362 during firing. In any case, if 4172 tissue stretch / displacement (d) is greater than or equal to a predetermined threshold, the control circuit adjusts 4173 the firing parameters of the surgical instrument 4600 to reduce tissue stretch / displacement (d). For example, the control circuit can cause the controller 4620 to reduce the speed of the firing motor drive assembly 4604 by, for example, reducing the current consumption of the firing motor 4602, for example, which reduces the advancement speed of the cutting member 4645. Additionally or alternatively, the control circuit can cause the controller 4620 to pause the firing motor 4602 for a predetermined period of time to reduce tissue stretch / displacement (d).

[0393] After firing, Figure 36C As shown, the jaws of the end effector 4642 are released and the stapled tissue is contracted due to the firing staples of the staple line 4363. Figure 36C Shown are a projected staple line length defined by distance (d1) and an actual staple line defined by distance (d3) which is less than distance (d1). The difference between distances d1, d2 represents the contraction / displacement distance (d').

[0394] Figure 38 4180 is a logic flow diagram of a process 4180 that depicts a control program or logic configuration for adjusting a recommended staple cartridge arrangement along a recommended surgical resection path. The process 4180 includes monitoring 4081 contraction / displacement of stapled tissue along the recommended surgical resection path after firing the staple cartridge of the recommended arrangement, and adjusting a subsequent staple cartridge position of the recommended arrangement along the recommended surgical resection path.

[0395] One or more aspects of process 4180 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4180 may be performed by a control circuit (e.g., Figure 2A4180) is performed by a control circuit 400 that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4180. Additionally or alternatively, one or more aspects of process 4180 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2C Furthermore, process 4180 may be performed by any suitable circuit having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0396] In various examples, control circuitry (e.g., control circuitry 132, 400, 410, 420, 602, 622, 2108, 4620) performing one or more aspects of process 4180 monitors 4181 for contraction / displacement of the stapled tissue along the recommended resection path 4312. Figure 36C In the example shown, staple line 4363 is deployed from a staple cartridge of staple cartridge arrangement 4317 into tissue between landmarks 4361 and 4362. When the jaws of end effector 4642 are released, the stapled tissue contracts / displaces a distance (d'). Distance (d') is the difference between distance (d1) representing the length of staple line 4361 recommended by arrangement 4317 and between landmarks 4361 and 4362 before firing, and distance (d3) representing the actual length of staple line 4363.

[0397] To avoid gaps between consecutive staple lines, the control circuit adjusts the subsequent staple cartridge positions of the recommended arrangement 4317 along the recommended surgical resection path 4312. For example, Figure 36C As shown, the initially recommended staple line 4368 is removed and replaced by an updated staple line 4369 that extends through or covers the gap defined by the distance (d'). In various aspects, tissue contraction / displacement (d') is monitored 4181 by using visualization data to track distortion in the structured light grid projected onto the tissue after the jaws of the end effector 4642 are released and / or tracking landmarks 4364, 4365, 4366, 4367 representing the positions of adjacent anatomical structures. Additionally or alternatively, the tissue contraction distance (d') can be monitored 4181 by tracking the position of the landmark 4362.

[0398] In various aspects, it may be desirable to use non-visual data from a non-visualization system to confirm visual data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108), and vice versa. In one example, the non-visualization system may include a ventilator that may be configured to measure non-visual data of the patient's lungs, such as volume, pressure, partial pressure of carbon dioxide (PCO2), partial pressure of oxygen (PO2), etc. Confirming visual data with non-visual data provides the clinician with greater confidence that the visual data derived from the visualization system is accurate. In addition, confirming visual data with non-visual data allows the clinician to better identify postoperative complications, as well as determine the overall efficiency of the organ, as will be described in more detail below. Confirmation may also be helpful for segmentectomies or complex lobectomies without fissures.

[0399] In various aspects, a clinician may need to remove a portion of a patient's organ to remove critical structures, such as a tumor and / or other tissue. In one example, the patient's organ may be a right lung. A clinician may need to remove a portion of a patient's right lung to remove unhealthy tissue. However, a clinician may not want to remove too much of a patient's lung during surgery to ensure that the function of the lung is not too compromised. The function of the lung can be assessed based on the peak lung volume per breath, which represents the peak lung capacity. In determining how much lung can be safely removed, the clinician is limited by a predetermined amount of reduction in peak lung volume, beyond which the lung will lose its vitality and require a total organ removal.

[0400] In at least one example, the surface area and / or volume of the lungs is estimated based on visualization data from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108). The surface area and / or volume of the lungs can be estimated at peak lung volume or peak lung volume per breath. In at least one example, the surface area and / or volume of the lungs can be estimated at multiple points throughout the inspiration / expiration cycle. In at least one aspect, before a portion of the lung is removed, the surface area and / or volume of the lungs determined by the visualization system can be correlated with the lung volume determined by the ventilator using visualization data and non-visualization data. For example, correlation data can be used to establish a mathematical relationship between the surface area and / or volume of the lungs derived from the visualization data and the lung volume determined by the ventilator. This relationship can be used to estimate the size of the portion of the lung that can be removed while maintaining the reduction in peak lung volume to a value less than or equal to a predetermined threshold value that maintains the viability of the lung.

[0401] Figure 39A logic flow diagram of a process 4750 for recommending surgical resection of a portion of an organ according to at least one aspect of the present disclosure is shown. Process 4750 is typically performed during a surgical procedure. Process 4750 may include recommending 4752 a portion of an organ to be resected based on visualization data from a surgical visualization system, wherein resection of the portion is configured to produce an estimated volume reduction of the organ. Process 4750 may further include determining 4754 a first value of a non-visualization parameter of the organ before resecting the portion, and determining 4756 a second value of the non-visualization parameter of the organ after resecting the portion. Additionally, in some examples, process 4750 may further include confirming 4758 the predetermined volume reduction based on the first value of the non-visualization parameter and the second value of the non-visualization parameter.

[0402] One or more aspects of process 4750 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4750 may be performed by a control circuit (e.g., Figure 2A 4750) is performed by a control circuit 400 that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4750. Additionally or alternatively, one or more aspects of process 4750 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2C Furthermore, one or more aspects of process 4750 may be performed by any suitable circuitry having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0403] In various aspects, the process 4750 may be performed by the computer-implemented interactive surgical system 2100 ( Figure 19 ), the computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include a remote server 2113. The control circuitry that performs one or more aspects of process 4750 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0404] Figure 41AA group of patient lungs 4780 are shown. In one embodiment, a clinician may utilize an imaging device 4782 to emit 4784 a pattern of light 4785, such as stripes, grid lines, and / or dots, onto the surface of the patient's right lung 4786 to enable determination of the topography or overall view of the surface of the patient's right lung 4786. The imaging device may be similar in various respects to imaging device 120 ( Figure 1 As described elsewhere herein, the projected light array can be used to determine the shape defined by the surface of the patient's right lung 4786 and / or the movement of the patient's right lung 4786 during surgery. In one embodiment, the imaging device 4782 can be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system, such as the surgical visualization system 100, can utilize the surface mapping logic 136 of the control circuitry 133 as described elsewhere herein to determine the topography or overall view of the surface of the patient's right lung 4786.

[0405] A clinician may provide a surgical system (such as surgical system 2100) with the type of surgery to be performed, such as a right upper lobectomy. In addition to providing the surgical procedure to be performed, the clinician may also provide the surgical system with the maximum desired volume of the organ to be removed during the surgical procedure. Based on the visualization data obtained from the imaging device 4782, the type of surgical procedure to be performed, and the maximum desired volume to be removed, the surgical system may recommend a resection path 4788 to remove a portion 4790 of the right lung that satisfies all of the clinician inputs. Other methods of recommending surgical resection paths are described elsewhere herein. The surgical system may consider any number of additional parameters in order to recommend a resection path 4788.

[0406] In various situations, it may be desirable to ensure that the volume of the organ removed produces the desired volume reduction of the patient's organ. To confirm that the volume removed produces the desired volume reduction, non-visual data from a non-visual system can be utilized. In one embodiment, a ventilator can be used to measure peak lung volume in a patient over time.

[0407] In at least one example, a clinician may utilize the surgical system 2100 during surgery to remove a lung tumor. 13A to 13E As described above, the control circuit can identify the tumor based on the visualization data and can recommend a surgical resection path that provides a safety margin around the tumor, as described above in conjunction with Figures 29 to 38As described. The control circuitry may further estimate the lung volume at peak lung volume. The ventilator may be used to measure peak lung volume prior to a surgical procedure. Using a predetermined mathematical correlation between the visually estimated lung volume and the lung volume as detected by the ventilator, the control circuitry is able to estimate the amount of peak lung volume reduction associated with removing a portion of the lung (including a safety margin of the tumor and peritumoral tissue). If the estimated amount of lung volume reduction exceeds a predetermined safety threshold, the control circuitry may alert the clinician and / or recommend a different surgical resection path that results in a smaller amount of lung volume reduction.

[0408] Figure 41C Graph 4800 showing a patient's peak lung volume measured over time. Prior to resection of the portion of the organ (t1), the ventilator may measure the peak lung volume. Figure 41C , at time t1 prior to resection of portion 4790, peak lung volume is measured to be 6 L. In the example above where the surgical procedure to be performed is a right upper lobectomy, the clinician may wish to remove only the volume of the patient's lung that produces a predetermined amount of volume reduction so that the patient's ability to breathe is not compromised. In one embodiment, for example, the clinician may wish to remove a portion that results in a reduction in the patient's peak lung volume of up to approximately 17%. Based on the surgical procedure and the desired volume reduction, the surgical system may recommend a surgical resection path 4788 that achieves removal of the lung portion while maintaining a peak lung volume value greater than or equal to 83% of the peak unresected lung volume.

[0409] Use Figure 41C Based on the ventilator data shown, a clinician can monitor the patient's peak lung volume over time, for example, before 4802 and after 4804 resection of a portion of the lung 4790. At time t2, the portion of the lung 4790 is resected along the recommended resection path 4788. As a result, the peak lung volume measured by the ventilator decreases. The clinician can use the ventilator data (peak lung volume before resection 4802 and peak lung volume after resection 4804) to confirm that the resected lung volume produces the desired reduction in lung volume. Figure 41C As shown, after the resection, the peak lung volume has dropped to 5 L, representing a decrease of approximately 17% in peak lung volume, which is approximately the same as the expected volume reduction. Using the ventilator data, the clinician has greater confidence that the actual volume reduction is consistent with the expected volume reduction achieved through the recommended surgical resection path 4788. In other embodiments, in the event of a discrepancy between the non-visualized data and the visualized data, such as a greater than expected decrease in peak lung volume (too much lung resection) or a smaller than expected decrease in peak lung volume (not enough lung resection), the clinician can determine whether appropriate action should be taken.

[0410] Now refer to Figure 41B, the patient's right lung 4792 is shown after resection 4790. After resection 4790, the clinician may inadvertently cause an air leak 4794, which causes air to leak into the space between the lung 4794 and the chest wall, resulting in a pneumothorax 4796. Due to the air leak 4794, as the right lung 4792 collapses, the patient's peak lung volume per breath will steadily decrease over time. Visualization data derived from a visualization system (e.g., visualization system 100, 160, 500, 2108) can be used to perform dynamic surface area / volume analysis of the lungs to detect air leaks by visually tracking changes in lung volume. The volume and / or surface area of the lungs can be visually tracked at one or more points during the inspiration / expiration cycle to detect volume changes indicative of an air leak 4794. In one embodiment, as described above, the projected light array from the imaging device 4782 can be used to monitor the movement of the patient's right lung 4786 over time, such as monitoring a decrease in size. In another embodiment, the surgical visualization system may utilize surface mapping logic, such as surface mapping logic 136, to determine the topography or panorama of the surface of the patient's right lung 4786 and monitor changes in the topography or panorama over time.

[0411] In one aspect, a clinician can utilize a non-visualization system, such as a ventilator, to confirm the reduction in volume detected by the visualization system. Figure 41C As described above, the patient's peak lung volume can be measured before 4802 and after resection of a portion of the lung to confirm that the desired volume reduction is consistent with the actual reduction in lung volume. In the example described above, in the event of an inadvertent air leak, the peak lung volume can steadily decrease over time 4806. In one instance, at time t2 immediately after resection of the portion, the clinician can record a decrease in peak lung volume from 6 L to 5 L, which is roughly consistent with the desired decrease in lung volume. After resection of the portion, the surgical visualization system can monitor the patient's lung volume over time. If the surgical visualization system determines that there is a volume change, the clinician can measure the peak lung volume again, for example, at time t3. At time t3, the clinician can record a decrease in peak lung volume from 5 L to 4 L, which confirms the data determined from the visualization system indicating that there may be an air leak in the right lung 4792.

[0412] In addition, the control circuitry can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, organ efficiency can be determined by comparing the visualization data with the difference in non-visualization data before and after resection of the portion. In one example, the visualization system can generate a resection path to reduce peak lung volume by 17%. The ventilator can be configured to measure peak lung volume before and after resection of the portion. Figure 41CIn the case shown, there is approximately a 17% decrease in peak lung volume (6 L to 5 L). Because the actual decrease in lung volume (17%) is close to a 1:1 ratio with the expected decrease in lung volume (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can generate a resection path to reduce peak lung volume by 17%. However, as an example, the ventilator can measure a decrease in peak lung volume that is greater than 17%, such as 25%. In this case, the clinician can determine that the lung is not functionally effective because resecting that portion of the lung resulted in a greater decrease in peak lung volume than expected.

[0413] Figure 40 A logic flow diagram of process 4760 is shown for estimating the volume reduction of an organ resulting from the removal of a selected portion of an organ, according to at least one aspect of the present disclosure. Process 4760 is similar in many respects to process 4750. However, unlike process 4750, process 4760 relies on a clinician to select or recommend a surgical resection path for removing a portion of an organ during a surgical procedure. Process 4760 includes receiving 4762 input from a user indicating a portion of the organ to be removed. Process 4760 further includes estimating 4764 the volume reduction of the organ resulting from the removal of the portion. In at least one example, the organ is a patient's lung, and the volume reduction estimated 4762 is the reduction in peak lung volume per breath of the patient's lung. The volume reduction corresponding to the removal of the portion can be estimated using visualization data from a surgical visualization system (e.g., visualization system 100, 160, 500, 2108). Process 4760 may further include determining 4766 a first value of a non-visualization parameter of the organ before resecting the portion, and determining 4768 a second value of the non-visualization parameter of the organ after resecting the portion. Finally, process 4760 may further include confirming 4768 the estimated volume reduction of the organ based on the first value of the non-visualization parameter and the second value of the non-visualization parameter.

[0414] One or more aspects of process 4760 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4760 may be performed by a control circuit (e.g., Figure 2A 4760) is performed by a control circuit 400 that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4760. Additionally or alternatively, one or more aspects of process 4760 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2CFurthermore, one or more aspects of process 4760 may be performed by any suitable circuitry having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0415] In various aspects, the process 4760 may be performed by the computer-implemented interactive surgical system 2100 ( Figure 19 ), the computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include a remote server 2113. The control circuitry that performs one or more aspects of process 4760 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0416] In one instance, a clinician may provide input to a surgical visualization system (such as surgical visualization system 2100) indicating a portion of an organ to be resected. In one instance, a clinician may draw a resection path on a virtual 3D construct of the organ (such as the virtual 3D construct generated 4104 during process 4100). In other instances, the visualization system may overlay a surgical layout plan described in more detail elsewhere herein, which may be in the form of a suggested treatment path. The suggested treatment path may be based on the type of surgical procedure being performed. In one embodiment, the suggested treatment path may recommend different starting points and different resection paths that the clinician may choose between resection paths 4146, 4147, 4148 similar to those described elsewhere herein. The recommended resection path may be determined by the visualization system to avoid certain critical structures such as arteries. The clinician may select the recommended resection path until the desired resection path for removing the portion of the organ is completed.

[0417] In one embodiment, a surgical visualization system may determine an estimated volume reduction for an organ based on the selected resection path. After resecting a predetermined portion along the resection path, the clinician may wish to use non-visual data to confirm that the actual volume reduction corresponds to the estimated volume reduction based on the visual data. In one embodiment, this confirmation can be accomplished using a similar procedure as described above for process 4750 in which the organ is a lung. Peak lung volume is measured before 4802 and after 4804 lung resection, and the change in peak lung volume is compared to determine the actual decrease in peak lung volume. In one example, a surgical visualization system may estimate a 17% decrease in peak lung volume based on the resection path recommended by the clinician. Before resection, the clinician may record a peak lung volume of 6 L (time t1). After resection, the clinician may record a peak lung volume of 5 L (time t2), which is approximately a 17% decrease in peak lung volume. Using this non-visual / ventilator data, the clinician has greater confidence that the actual volume reduction is consistent with the estimated volume reduction. In other cases, where there is a discrepancy between the non-visualized and visualized data, such as a larger than expected drop in peak lung volume (over-resection) or a smaller than expected drop in peak lung volume (under-resection), the clinician can determine whether appropriate action should be taken.

[0418] In addition, the control circuitry can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, organ efficiency can be determined by comparing the visualization data to the difference in non-visualization data before and after resection of the portion. In one example, the visualization system can estimate a 17% reduction in peak lung volume based on the clinician's desired resection path. The ventilator can be configured to measure peak lung volume before and after resection of the portion. Figure 41C In the case shown, there is approximately a 17% decrease in peak lung volume (6 L to 5 L). Because the actual decrease in lung volume (17%) is close to a 1:1 ratio with the estimated decrease in lung volume (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can estimate a 17% decrease in peak lung volume based on the clinician's desired resection path. However, as an example, the ventilator can measure a decrease in peak lung volume that is greater than 17%, such as 25%. In this case, the clinician can determine that the lung is not functionally effective because resecting that portion of the lung resulted in a greater than expected decrease in peak lung volume.

[0419] As described above with respect to processes 4750, 4760, a clinician can use non-visual data (e.g., by using a ventilator to measure peak lung volume before and after removing a portion of a lung) to corroborate visual data. Another example of using non-visual data to corroborate visual data is through capnography.

[0420] Figure 42 A graph 4810 is shown measuring the partial pressure of carbon dioxide (PCO2) exhaled by a patient over time. In other cases, the partial pressure of oxygen (PO2) exhaled by a patient may be measured over time. Graph 4810 shows PCO2 measured before 4812, immediately after 4814, and one minute after 4816. Figure 42 In the example described above, prior to resection 4812, the PCO2 is measured to be approximately 40 mmHg (at time t1). In the example described above, where the surgical procedure to be performed is a right upper lobectomy, the visualization system may expect or estimate a 17% reduction in lung volume. The PCO2 level measured by the ventilator can be used to confirm this expected or estimated volume reduction.

[0421] Use Figure 42 Using the ventilator data shown, a clinician can monitor the patient's PCO2 over time, for example, before 4812 and immediately after 4814 resection of a portion of the lung 4790. At time t2, the portion of the lung 4790 has been resected, and therefore, the PCO2 measured by the ventilator may have dropped 4818. The clinician can use the ventilator data (pre-resection 4812 PCO2 (at t1) and post-resection 4814 PCO2 (at t2)) to confirm that the actual reduction in lung volume is consistent with the estimated or expected reduction in lung volume. Figure 42 As shown, immediately following resection 4814 of portion 4790, PCO2 decreases 4812, which can be measured as approximately a 17% decrease in PCO2 (approximately 33.2 mmHg). Using this non-visualized / ventilator data, the clinician has greater confidence that the actual reduction in lung volume is consistent with the expected or estimated reduction in lung volume.

[0422] In other cases, the clinician can utilize the non-visualized / PCO2 data to determine the difference when compared to the visualized data. In one case, immediately after resection 4814 of portion 4790, at time t2, a PCO2 can be measured at 4820 that is higher than the PCO2 measured before resection 4812. The increase in PCO2 may be the result of inadvertent bronchial obstruction during surgery, resulting in a buildup of CO2 in the patient. In another case, immediately after resection 4814 of portion 4790, at time t2, a PCO2 can be measured at 4822 that is lower than the PCO2 measured before resection 4812 and lower than expected. The decrease in PCO2 may be the result of inadvertent occlusion of a blood vessel during surgery, resulting in less O2 being delivered to the body and, therefore, less CO2 being produced. In either case, the clinician can take appropriate measures to remedy the situation.

[0423] The change in PCO2 can also be measured at a time other than immediately after resection 4814, for example, one minute after resection 4816 (e.g., at time t3). At time t3, other body functions (such as the kidneys) compensate for the change in PCO2 due to resection. In this case, the PCO2 can be measured to be approximately 40 mmHg, or approximately the same as the measurement before resection 4812. At time t3, the difference between the measured PCO2 and the PCO2 before resection 4812 can indicate the inadvertent obstruction discussed above. For example, at time t3, the PCO2 can be measured 4824 to be higher than before resection 4812, indicating a possible inadvertent obstruction of the bronchi, or the PCO2 can be measured 4826 to be lower than before resection 4812, indicating a possible inadvertent obstruction of the blood vessels.

[0424] In addition, the control circuitry can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, organ efficiency can be determined by comparing the visualization data to the difference in non-visualization data before and after resection of the portion. In one example, the visualization system can estimate a 17% reduction in lung volume based on the clinician's desired resection path. The ventilator can be configured to measure PCO2 before and after resection of the portion. Figure 42 In the illustrated embodiment, immediately after resection 4814PCO2 drops by about 17%. Since the PCO2 drop (17%) is close to 1:1 with the estimated lung capacity drop (17%), the clinician can determine that the lung is functionally effective. In another example, the visualization system can estimate a 17% reduction in lung capacity based on the clinician's desired resection path. However, as an example, a ventilator can measure a PCO2 drop greater than 17% (such as 25%). In this case, the clinician can determine that the lung is not functionally effective because resection of the portion of the lung causes a greater PCO2 drop than expected.

[0425] In addition to the peak lung volume and PCO2 measurements mentioned above, other non-visual parameters can be utilized, including blood pressure or EKG data. EKG data will provide approximate frequency data about arterial deformation. This frequency data, which has surface geometry variations within a similar frequency range, can help identify key vascular structures.

[0426] As described above, it may be desirable to utilize non-visual data from a non-visualization system to confirm visual data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). In the above examples, the non-visual data provides a means for confirming visual data after a portion of an organ has been resected. In some cases, it may be desirable to supplement visual data with non-visual data before resecting a portion of an organ. In one example, the non-visual data can be used along with the visual data to help determine features of the organ to be operated on. In one aspect, the features can be abnormalities in the tissue of the organ that may not be suitable for cutting. The non-visual data and the visual data can help inform the surgical visualization system and the clinician of areas to avoid when planning a path for organ resection. This can be helpful for segmentectomies or complex lobectomies without fissures.

[0427] Figure 43 A logic flow diagram is shown of a process 4850 for detecting a tissue abnormality based on visualization data and non-visualization data, in accordance with at least one aspect of the present disclosure. Process 4850 is typically performed during a surgical procedure. Process 4850 may include receiving 4852 first visualization data from a surgical visualization system in a first state of an organ and determining 4854 a first value for a non-visualization parameter of the organ in the first state. Process 4850 may further include receiving 4856 second visualization data from the surgical visualization system in a second state of the organ and determining 4858 a second value for the non-visualization parameter of the organ in the second state. The process may also include detecting 4860 a tissue abnormality based on the first visualization data, the second visualization data, the first value for the non-visualization parameter, and the second value for the non-visualization parameter.

[0428] One or more aspects of process 4850 may be performed by one or more of the control circuits described in this disclosure (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620). In at least one example, one or more aspects of process 4850 may be performed by a control circuit (e.g., Figure 2A 4850) is performed by a control circuit 400 that includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4850. Additionally or alternatively, one or more aspects of process 4850 may be performed by a combinational logic circuit (e.g., Figure 2B control circuit 410) and / or sequential logic circuits (e.g., Figure 2CFurthermore, one or more aspects of process 4850 may be performed by any suitable circuitry having any suitable hardware and / or software components, which may be located in or associated with various suitable systems described herein.

[0429] In various aspects, the process 4850 may be performed by the computer-implemented interactive surgical system 2100 ( Figure 19 ), the computer-implemented interactive surgical system includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104, which may include a remote server 2113. The control circuitry that performs one or more aspects of process 4850 may be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0430] Figure 44A The right lung 4870 of a patient is shown in a first state 4862. In one example, the first state 4862 can be a contracted state. In another example, the first state 4862 can be a collapsed state. An imaging device 4872 is shown inserted through a cavity 4874 in the patient's chest wall 4876. The clinician can use the imaging device 4872 to emit 4880 a pattern 4882 of light onto the surface of the right lung 4870, such as stripes, grid lines, and / or dots, to enable the topography or overall view of the surface of the patient's right lung 4870 to be determined. The imaging device can be similar in various respects to the imaging device 120 ( Figure 1 ). As described elsewhere herein, the projected light array is used to determine a shape defined by the surface of the patient's right lung 4870, and / or the movement of the patient's right lung 4870 during surgery. In one embodiment, the imaging device 4782 may be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system, such as the surgical visualization system 100, may utilize the surface mapping logic 136 of the control circuitry 133, as described elsewhere herein, to determine a topography or panorama of the surface of the patient's right lung 4786. In the first state 4862 of the right lung 4870, the ventilator may be used to measure a parameter of the right lung 4870, such as a first state pressure (P1, or positive end expiratory pressure (PEEP)) or a first state volume (V1).

[0431] Figure 44BThe right lung 4870 of the patient is shown in a second state 4864. In one example, the second state 4864 can be a partially inflated state. In another example, the second state 4864 can be a fully inflated state. The imaging device 4872 can be configured to continue emitting 4880 a pattern of light 4882 onto the surface of the lung 4870, thereby enabling determination of a topography or panorama of the surface of the patient's right lung 4870 in the second state 4864. In the second state 4864 of the right lung 4870, the ventilator can be used to measure parameters of the right lung 4870, such as a second state pressure (P2) that is greater than the first state 4862 pressure P1 and a second state volume (V2) that is greater than the first state 4862 volume V1.

[0432] Based on the surface topography determined from the surgical visualization system and the imaging device 4872, and the non-visualized data determined from the ventilator (pressure / volume), the surgical visualization system can be configured to determine tissue abnormalities of the right lung 4870. In one example, in the first state 4862, the imaging device 4872 can determine the first state 4662 topography of the right lung 4870 (at Figure 44A shown in and Figure 44C ), and the ventilator can determine the first state pressure / volume. In the second state 4864, the imaging device 4872 can determine the second state 4864 topography of the right lung 4870 (in Figure 44B shown in and Figure 44D ), and the ventilator can determine a second state pressure / volume that is greater than the first state pressure / volume due to partial or complete expansion of the lung. Based on the known pressure / volume increase, the visualization system can be configured to monitor the topographical changes of the right lung 4870 in response to the known pressure / volume increase. In one aspect, this pressure / volume measurement from the ventilator can be correlated with the surface deformation of the right lung 4870 to identify diseased areas within the lung, thereby helping to inform stapler placement.

[0433] In one aspect, reference Figure 44B and Figure 44D, where pressure increases from P1 to P2 (and volume increases from V1 to V2), the surface topography determined from structured light 4880 has changed compared to first state 4862. In one example, pattern 4882 of light can be dots, and as lung size increases, the dots become spaced apart by a certain distance. In another example, pattern 4882 of light can be grid lines, and as lung size increases, the grid lines become spaced apart or contoured. Based on the known pressure increase, the imaging device can identify areas 4886 that have not changed based on the known pressure and volume increases. For example, if imaging device 4872 emits pattern 4882 of grid lines and dots onto the surface of right lung 4870, the visualization system can be configured to monitor the contours of the grid lines and the positioning of the dots relative to each other for the known pressure / volume increase. If the visualization system notices irregularities in the spacing of the dots or the positioning and curvature of the grid lines, the visualization system can determine that these areas correspond to potential abnormalities in the tissue, such as areas where critical structures 4884 (such as a tumor) may be located, or subsurface voids 4886. In one embodiment, referring to process 4100 , where process 4100 identifies 4105 the anatomical structure of at least a portion of an anatomical organ relevant to a surgical procedure, process 4100 may identify anomalies as described above and superimpose these anomalies onto a 3D construct.

[0434] In one example, a patient may have emphysema, a lung disease that causes shortness of breath. In people with emphysema, the air sacs in the lungs (alveoli) are damaged, and over time, the inner walls of the air sacs weaken and rupture, creating larger air spaces instead of many small air spaces. This reduces the internal surface area of the lungs for O2 / CO2 exchange, thus reducing the amount of oxygen reaching the bloodstream. In addition, the damaged alveoli do not function properly, old air is trapped, and there is no room for fresh, oxygen-rich air to enter. The empty spaces within the lungs of patients with emphysema represent areas with less tissue thickness, thus affecting the suturing results in that area. The tissue is also weakened, which causes the alveoli to rupture and is less able to hold the nails that pass through them.

[0435] As a lung with emphysema expands and contracts, areas with subsurface voids will deform differently as pressure changes compared to healthy tissue. Using the above-described process 4850, these weak tissue areas with subsurface voids can be detected to inform clinicians that they should avoid suturing through these areas, which can reduce the likelihood of postoperative air leaks. The tissue deformation capabilities of this process 4850 will allow these differences to be detected, thereby guiding the surgeon when placing the stapler.

[0436] In a second example, a patient may have cancer. Prior to surgery, the tumor may have been irradiated, which damages the tissue and surrounding tissue. Irradiation changes the properties of the tissue, typically making it harder and less compressible. If the surgeon needs to perform sutures on this tissue, the change in tissue hardness should be considered when selecting the type of staple reload (e.g., harder tissue will require a higher-profile staple).

[0437] As the lung expands and contracts, areas with stiffer tissue will deform differently than healthy tissue because the lung is less compliant in these areas. The tissue deformation capabilities of this process 4850 will allow these differences to be detected, thereby guiding the surgeon when placing staplers and selecting cartridge / reloading colors.

[0438] In another aspect, a memory (such as memory 134) may be configured to store lung surface topography at known pressures and volumes. In this case, an imaging device, such as imaging device 4872, may emit a pattern of light to determine the topography of the patient's lung surface at a first known pressure or volume. A surgical system, such as surgical system 2100, may be configured to compare the first determined topography at the known first pressure or volume with the topography stored in memory 134 at the given first pressure or volume. Based on this comparison, the visualization system may be configured to indicate potential abnormalities in the tissue only at a single state. The visualization system may record these potentially abnormal areas and continue to determine the topography of the patient's lung surface at a second known pressure or volume. The visualization system may compare the second determined surface topography with the topography stored in memory at the second given pressure or volume and the topography determined at the first known pressure or volume. If the visualization system determines a potential abnormal area that overlaps with the first determined potential abnormal area, the visualization system may be configured to indicate the overlapping area as a potential abnormality with greater confidence based on the comparison between the first and second known pressures or volumes.

[0439] In addition to the above, PO2 measurements from a ventilator can be compared to inflated lung volumes (such as V2) and deflated lung volumes (such as V1). Volume comparisons can utilize EKG data to compare inspiration and expiration, which can be compared to blood oxygenation. This can also be compared to anesthetic gas exchange measurements to determine the relationship between respiratory volume, oxygen uptake, and sedation. In addition, EKG data can provide approximate frequency data on arterial deformation. This frequency data, which has surface geometry changes in a similar frequency range, can help identify key vascular structures.

[0440] In another embodiment, the current tracking / surgical information can be compared with a preoperative planned simulation. In challenging or high-risk surgeries, the clinician can utilize preoperative patient scans to simulate the surgical approach. This data set can be compared with real-time measurements in a display (such as display 146) to help the surgeon follow a specific preoperative plan based on a training run. This will require the ability to match fiducial landmarks between the preoperative scan / simulation and the current visualization. One approach can simply use boundary tracking of the object. Insights into how the current device-tissue interaction compares to previous interactions (per patient) or expected interactions (database or past patients) to perform tissue type discrimination, relative tissue deformation assessment, or subsurface structural differences can be stored in a memory such as memory 134.

[0441] In one embodiment, the surface geometry can be a function of the tool position. When no change in surface geometry is measured for each change in tool position, a surface reference can be selected. As the tool interacts with the tissue and deforms the surface geometry, the surgical system can calculate the change in surface geometry as a function of the tool position. For a given change in tool position when in contact with the tissue, the change in tissue geometry may be different in areas with subsurface structures (such as critical structures 4884) than in areas without such structures (such as subsurface voids 4886). In an example such as thoracic surgery, this may be above the airway rather than just in the parenchyma. The surgical system can use the surgical visualization system to calculate a running average of the change in tool position relative to the change in surface geometry for a given patient, thereby giving patient-specific differences, or the value can be compared with a second set of previously collected data.

[0442] Exemplary clinical applications

[0443] The various surgical visualization systems disclosed herein may be used in one or more of the following clinical applications.The following clinical applications are non-exhaustive and merely illustrative applications for one or more of the various surgical visualization systems disclosed herein.

[0444] The surgical visualization system disclosed herein can be used in a variety of different types of surgeries, for example, in different medical specialties such as urology, gynecology, oncology, colorectal surgery, thoracic surgery, bariatrics / gastroenterology, and hepato-pancreatico-biliary surgery (HPB). For example, in urological surgery (such as a prostatectomy), ureters can be detected in fat, or connective tissue and / or nerves can be detected in fat. For example, in gynecological oncology surgery (such as a hysterectomy), and in colorectal surgery (such as a low anterior resection (LAR) surgery), ureters can be detected in fat and / or connective tissue. For example, in thoracic surgery (such as a lobectomy), blood vessels can be detected in the lungs or connective tissue, and / or nerves can be detected in connective tissue (e.g., esophagostomy). In bariatric surgery, blood vessels can be detected in fat. For example, in HPB surgery (such as hepatectomy or pancreatectomy), blood vessels can be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts can be detected in thin-walled (liver or pancreas) tissue.

[0445] In one example, a clinician may want to remove an endometrial fibroid. Based on a preoperative magnetic resonance imaging (MRI) scan, the clinician may know that the endometrial fibroid is located on the surface of the intestine. Therefore, the clinician may want to know during surgery which tissues constitute part of the intestine and which tissues constitute part of the rectum. In such a case, a surgical visualization system as disclosed herein can indicate the different types of tissue (intestine vs. rectum) and convey this information to the clinician via the imaging system. In addition, the imaging system can determine the proximity of the surgical device to the selected tissue and communicate this proximity. In such a case, the surgical visualization system can provide increased surgical efficiency without serious complications.

[0446] In another example, a clinician (e.g., a gynecologist) may stay away from certain anatomical areas to avoid getting too close to critical structures, and therefore, the clinician may not be able to remove, for example, all of the endometriosis. A surgical visualization system as disclosed herein may enable a gynecologist to reduce the risk of getting too close to critical structures, allowing the gynecologist to get close enough with a surgical device to remove all of the endometriosis, which may improve patient outcomes (democratizing surgery). Such a system may enable a surgeon to "keep moving" during a surgical procedure rather than repeatedly stopping and restarting in order to identify areas to avoid, for example, particularly during the application of therapeutic energy such as ultrasound or electrosurgical energy. In gynecological applications, the uterine artery and ureter are important critical structures, and given the presence and / or thickness of the tissue involved, the system may be particularly useful for hysterectomy and endometriosis surgery.

[0447] In another example, a clinician may risk dissecting a vessel too close together, and thus potentially affecting the blood supply to lobes other than the target lobe. Furthermore, anatomical variations from patient to patient may result in dissecting vessels (e.g., branches) that affect different lobes depending on the patient. A surgical visualization system as disclosed herein can enable identification of the correct vessel at the desired location, enabling the clinician to perform the dissection with appropriate anatomical certainty. For example, the system can confirm that the correct vessel is in the correct location, and the clinician can then safely divide the vessel.

[0448] In another example, due to uncertainty in the anatomy of a vessel, a clinician may perform multiple dissections before reaching the optimal location. However, it is desirable to perform the dissection at the optimal location first, as more dissections can increase the risk of bleeding. A surgical visualization system, as disclosed herein, can minimize the number of dissections by indicating the correct vessel and optimal location for dissection. For example, the ureters and cardinal ligaments are densely packed and present unique challenges during dissection. In such cases, minimizing the number of dissections may be particularly desirable.

[0449] In another example, a clinician (e.g., a surgical oncologist removing cancerous tissue) may wish to know the identification of key structures, the location of the cancer, the stage of the cancer, and / or an assessment of tissue health. Such information goes beyond what the clinician can see with the "naked eye." A surgical visualization system as disclosed herein can determine such information intraoperatively and / or communicate such information to the clinician to enhance intraoperative decision-making and improve surgical outcomes. In some cases, the surgical visualization system can be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, for example, using an endoscope or an exoscope.

[0450] In another example, a clinician (e.g., a surgical oncologist) may want to turn off one or more alerts about the proximity of surgical tools to one or more critical structures to avoid being too conservative during surgery. In other cases, the clinician may want to receive certain types of alerts such as tactile feedback (e.g., vibration / beep) to indicate proximity and / or a "no-fly zone" to keep sufficiently away from one or more critical structures. For example, a surgical visualization system as disclosed herein can provide flexibility based on the clinician's experience and / or the desired aggressiveness of the surgery. In such cases, the system provides a balance between "knowing too much" and "knowing enough" to anticipate and avoid critical structures. The surgical visualization system can help plan the next steps during surgery.

[0451] Various aspects of the subject matter described herein are set forth in the following numbered examples:

[0452] Example 1. A surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ, the surgical system comprising at least one imaging device and a control circuit, the control circuit being configured to identify anatomical structures associated with the surgical procedure based on visualization data from the at least one imaging device, recommend a surgical resection path for removing a portion of the anatomical organ by the surgical instrument, wherein the surgical resection path is determined based on the anatomical structure, and present parameters of the surgical instrument based on the surgical resection path.

[0453] Example 2. A surgical system according to Example 1, wherein the parameters are presented along the surgical resection path.

[0454] Example 3. A surgical system according to Example 1 or 2, wherein the surgical resection path is superimposed on a 3D structure of at least a portion of the anatomical organ.

[0455] Example 4. A surgical system according to Example 3, wherein the parameters are superimposed along the surgical resection path.

[0456] Example 5. A surgical system according to any one of Examples 1 to 4, wherein the surgical instrument is a surgical stapler, and wherein the parameters include at least one of staple cartridge size, staple cartridge color, staple cartridge type, and staple cartridge length.

[0457] Example 6. A surgical system according to any one of Examples 1 to 5, wherein the parameters of the surgical instrument are functional parameters, and the functional parameters include at least one of a closing force (FTC) parameter, a firing force (FTF) parameter, a firing speed parameter and a closing speed parameter.

[0458] Example 7. A surgical system according to any one of Examples 1 to 6, wherein the parameter is selected based on at least one tissue thickness along the surgical resection path.

[0459] Example 8. A surgical system according to any one of Examples 1 to 7, wherein the control circuit is further configured to superimpose tissue parameters along the surgical resection path.

[0460] Example 9. A surgical system according to Example 8, wherein the tissue parameters include at least one of tissue thickness, tissue type, and volumetric results of the surgical resection path.

[0461] Example 10. A surgical system for use with a surgical instrument in a surgical procedure performed on an anatomical organ, the surgical system comprising at least one imaging device and a control circuit, the control circuit being configured to identify anatomical structures associated with the surgical procedure based on visualization data from the at least one imaging device, recommend a surgical resection path for removing a portion of the anatomical organ via the surgical instrument, wherein the surgical resection path is determined based on the anatomical structure, and adjust parameters of the surgical instrument based on the surgical resection path.

[0462] Example 11. A surgical system according to Example 10, wherein the ...

Claims

1. A surgical system for use with a surgical stapling instrument in a surgical procedure performed on an anatomical organ, the surgical system comprising: at least one imaging device; and A control circuit configured to: a recommended surgical resection path for removing a portion of the anatomical organ by the surgical stapling instrument; recommending a staple cartridge placement along the surgical resection path; as well as Displacement of the tissue along a recommended surgical resection path due to deployment of staple lines from a staple cartridge of the staple cartridge arrangement into the tissue is monitored.

2. The surgical system of claim 1 , wherein: The displacement is detected during firing of the surgical stapling instrument, and wherein the control circuit is further configured to adjust a parameter of the surgical stapling instrument to reduce the displacement.

3. The surgical system of claim 2, wherein: The control circuit is configured to adjust the firing speed of the surgical stapling instrument.

4. The surgical system of claim 1 , wherein: After the staple line is deployed from a staple cartridge of the staple cartridge arrangement into the tissue, the displacement is detected after the jaws of the surgical stapling instrument are released, and wherein the control circuit is further configured to adjust a subsequent staple cartridge position of the staple cartridge arrangement to compensate for the displacement.

Images