Surgical systems for proposing and corroborating organ portion removals

Abstract

To provide certain hidden structures, physical contours, and / or dimensions in three-dimensional space which provide information that cannot be recognized intraoperatively by certain imaging systems.SOLUTION: The surgical system includes a surgical visualization system and control circuitry configured to estimate, on the basis of visualization data from the surgical visualization system, at least one of a surface area and a volume of a lung with no portion resected, and estimate, on the basis of the estimated at least one of the surface area and the volume of the lung, a first maximum lung volume of the lung with no portion resected, estimate a second maximum lung volume of the lung after a portion of the lung is resected by a resection path determined on the basis of the visualization data from the surgical visualization system, and suggest the resection path when a ratio of the second maximum lung volume to the first maximum lung volume is greater than or equal to a predetermined value.SELECTED DRAWING: None

Description

Background Art

[0001] Surgical systems often incorporate an imaging system that enables a clinician(s) to view the surgical site and / or one or more portions thereof on one or more displays such as a monitor. The display(s) may be local to the surgical theater and / or remote. The imaging system can include a scope with a camera that views the surgical site and transmits a view to a display that the clinician can view. Scopes can include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, esophagogastroduodenoscopes, enteroscopes, esophagogastroduodenoscopes (stomach cameras), endoscopes, laryngoscopes, nasopharyngo-ureteroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and external scopes. The imaging system may be limited by information that can recognize a clinician(s) and / or communicate to a clinician. For example, certain hidden structures, physical contours, and / or dimensions within a three-dimensional space may not be recognizable during surgery with a particular imaging system. Additionally, certain imaging systems may not be able to communicate and / or transmit certain information to a clinician(s) during surgery.

Summary of the Invention

Means for Solving the Problems

[0002] In one general aspect, a surgical system for use in surgery is disclosed. The surgical system includes a surgical visualization system and a control circuit configured to propose a portion of an organ to be resected, determine a first value of a non-visualization parameter of the organ before resection of the portion, and determine a second value of the non-visualization parameter of the organ after resection of the portion based on visualization data from the surgical visualization system. The resection of the portion is configured to result in an estimated volume reduction of the organ.

[0003] In another general aspect, a surgical system for use in a surgical operation is disclosed. This surgical system includes a surgical visualization system and a control circuit configured to receive, from the user, an input indicating a portion of an organ to be resected based on visualization data from the surgical visualization system, and to estimate a reduction in the volume of the organ by removing the portion.

[0004] In yet another general aspect, a surgical system for use in a surgical operation is disclosed. This surgical system includes a surgical visualization system and a control circuit configured to receive first visualization data of an organ in a first state from the surgical visualization system, determine a first value of a non-visualization parameter of the organ in the first state, receive second visualization data of the organ in a second state from the surgical visualization system, determine a second value of the non-visualization parameter of the organ in the second state, and detect tissue abnormalities based on the first visualization data, the second visualization data, the first value of the non-visualization parameter, and the second value of the non-visualization parameter.

Brief Description of the Drawings

[0005] The novel features of the various aspects are specifically set forth in the appended "Claims". However, the aspects described can be best understood by reference to the following description taken in conjunction with the accompanying drawings, both as to the organization and the method of operation.

Figure 1

Figure 2

Figure 2A

Figure 2B

Figure 2C

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 13E

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25A

Figure 25B

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32A

Figure 32B

Figure 32C

Figure 32D

Figure 33

Figure 34

Figure 35

Figure 36A

Figure 36B

Figure 36C

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41A

Figure 41B

Figure 41C

Figure 42

Figure 43

Figure 44A

Figure 44B

Figure 44C

Figure 44D

DETAILED DESCRIPTION OF THE INVENTION

[0006] The applicant of the present application owns the following co-pending U.S. patent applications, each of which is hereby incorporated by reference in its entirety. · Attorney Docket No. END9228USNP1 / 190580-1M, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY"; · Attorney Docket No. END9227USNP1 / 190579-1, entitled "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM"; · Attorney Docket No. END9226USNP1 / 190578-1, entitled "SURGICAL SYSTEM CONTROL BASED ON MULTIPLE SENSED PARAMETERS"; · Attorney Docket No. END9225USNP1 / 190577-1, entitled "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS"; · Attorney Docket No. END9224USNP1 / 190576-1, entitled "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS"; · Attorney Docket No. END9223USNP1 / 190575-1, entitled "SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA"; · Entitled "SURGICAL SYSTEMS FOR GENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED", Attorney Docket No. END9222USNP1 / 190574-1; · Entitled "SURGICAL SYSTEM FOR OVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN", Attorney Docket No. END9221USNP1 / 190573-1; · Entitled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE", Attorney Docket No. END9219USNP1 / 190571-1; · Entitled "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT", Attorney Docket No. END9218USNP1 / 190570-1; · Entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS", Attorney Docket No. END9217USNP1 / 190569-1; · Entitled "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM", Attorney Docket No. END9216USNP1 / 190568-1.

[0007] The applicant of the present application owns the following U.S. patent applications filed on March 15, 2019, each of which is hereby incorporated by reference in its entirety. · Entitled "INPUT CONTROLS FOR ROBOTIC SURGERY", U.S. Patent Application No. 16 / 354,417; · U.S. Patent Application No. 16 / 354,420, titled "DUAL MODE CONTROLS FOR ROBOTIC SURGERY"; · U.S. Patent Application No. 16 / 354,422, titled "MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY"; · U.S. Patent Application No. 16 / 354,440, titled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY"; · U.S. Patent Application No. 16 / 354,444, titled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE"; · U.S. Patent Application No. 16 / 354,454, titled "ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS"; · U.S. Patent Application No. 16 / 354,461, titled "SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS"; · U.S. Patent Application No. 16 / 354,470, titled "SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS"; · U.S. Patent Application No. 16 / 354,474, titled "ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES"; · U.S. Patent Application No. 16 / 354,478, titled "ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK, and · U.S. Patent Application No. 16 / 354,481, titled "JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS".

[0008] The applicant of this application also owns the following U.S. patent applications filed on September 11, 2018, and the entire contents of each of these are incorporated herein by reference. · U.S. Patent Application No. 16 / 128,179, titled "SURGICAL VISUALIZATION PLATFORM"; · U.S. Patent Application No. 16 / 128,180, titled "CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE"; · U.S. Patent Application No. 16 / 128,198, titled "SINGULAR EMR SOURCE EMITTER ASSEMBLY"; · U.S. Patent Application No. 16 / 128,207, titled "COMBINATION EMITTER AND CAMERA ASSEMBLY"; · U.S. Patent Application No. 16 / 128,176, titled "SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES"; · U.S. Patent Application No. 16 / 128,187, titled "SURGICAL VISUALIZATION OF MULTIPLE TARGETS"; · U.S. Patent Application No. 16 / 128,192, titled "VISUALIZATION OF SURGICAL DEVICES"; · U.S. Patent Application No. 16 / 128,163, titled "OPERATIVE COMMUNICATION OF LIGHT"; · U.S. Patent Application No. 16 / 128,197, titled "ROBOTIC LIGHT PROJECTION TOOLS"; · U.S. Patent Application No. 16 / 128,164, titled "SURGICAL VISUALIZATION FEEDBACK SYSTEM"; · U.S. Patent Application No. 16 / 128,193, entitled "SURGICAL VISUALIZATION AND MONITORING"; · U.S. Patent Application No. 16 / 128,195, entitled "INTEGRATION OF IMAGING DATA"; · U.S. Patent Application No. 16 / 128,170, entitled "ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS"; · U.S. Patent Application No. 16 / 128,183, entitled "SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS"; · U.S. Patent Application No. 16 / 128,172, entitled "ROBOTIC SYSTEM WITH SEPARATE PHOTOACOUSTIC RECEIVER"; · U.S. Patent Application No. 16 / 128,185, entitled "FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION".

[0009] The applicant of the present application also owns the following U.S. patent applications filed on March 29, 2018, and the entire contents of each of these are incorporated herein by reference. · U.S. Patent Application No. 15 / 940,627, entitled "DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", currently U.S. Patent Application Publication No. 2019 / 0201111; · U.S. Patent Application No. 15 / 940,676, entitled "AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", currently U.S. Patent Application Publication No. 2019 / 0201142; · U.S. Patent Application No. 15 / 940,711, entitled "SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", currently U.S. Patent Application Publication No. 2019 / 0201120; · U.S. Patent Application No. 15 / 940,722, entitled "CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY", currently published as U.S. Patent Application Publication No. 2019 / 0200905.

[0010] The applicant of this application owns the following U.S. patent applications filed on December 4, 2018, and the disclosures of each of these are hereby incorporated by reference in their entirety into this specification. · U.S. Patent Application No. 16 / 209,395, entitled "METHOD OF HUB COMMUNICATION", currently published as U.S. Patent Application Publication No. 2019 / 0201136; · U.S. Patent Application No. 16 / 209,403, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB", currently published as U.S. Patent Application Publication No. 2019 / 0206569; · U.S. Patent Application No. 16 / 209,407, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", currently published as U.S. Patent Application Publication No. 2019 / 0201137; · U.S. Patent Application No. 16 / 209,416, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS", currently published as U.S. Patent Application Publication No. 2019 / 0206562; · U.S. Patent Application No. 16 / 209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", currently published as U.S. Patent Application Publication No. 2019 / 0200981; · U.S. Patent Application No. 16 / 209,427, currently published as U.S. Patent Application Publication No. 2019 / 0208641, titled "METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES"; · U.S. Patent Application No. 16 / 209,433, currently published as U.S. Patent Application Publication No. 2019 / 0201594, titled "METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB"; · U.S. Patent Application No. 16 / 209,447, currently published as U.S. Patent Application Publication No. 2019 / 0201045, titled "METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB"; · U.S. Patent Application No. 16 / 209,453, currently published as U.S. Patent Application Publication No. 2019 / 0201046, titled "METHOD FOR CONTROLLING SMART ENERGY DEVICES"; · U.S. Patent Application No. 16 / 209,458, currently published as U.S. Patent Application Publication No. 2019 / 0201047, titled "METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE"; · U.S. Patent Application No. 16 / 209,465, currently published as U.S. Patent Application Publication No. 2019 / 0206563, titled "METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION"; · U.S. Patent Application No. 16 / 209,478, now published as U.S. Patent Application Publication No. 2019 / 0104919, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE"; · U.S. Patent Application No. 16 / 209,490, now published as U.S. Patent Application Publication No. 2019 / 0206564, entitled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION"; · U.S. Patent Application No. 16 / 209,491, now published as U.S. Patent Application Publication No. 2019 / 0200998, entitled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS".

[0011] Before describing in detail various aspects of the surgical visualization platform, it should be noted that the exemplary embodiments are not limited in their application or use to the details of the structure and arrangement of the components illustrated in the accompanying drawings and specification. The exemplary embodiments may be implemented in or incorporated into other aspects, variations, and modifications, and may be practiced or carried out in various ways. Further, unless otherwise specified, the terms and expressions employed herein are selected for the purpose of describing the exemplary embodiments for the convenience of the reader and are not intended to be limiting. Additionally, it should be understood that one or more of the aspects, expressions of aspects, and / or embodiments described below can be combined with any one or more of the other aspects, expressions of aspects, and / or embodiments described below.

[0012] Surgical Visualization System The present disclosure is directed to a surgical visualization platform that utilizes "digital surgery" to obtain additional information regarding a patient's anatomical structure and / or surgery. The surgical visualization platform is further configured to communicate data and / or information to one or more clinicians in a useful manner. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomical structure and / or surgery.

[0013] "Digital surgery" can include robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analysis for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and the like. The various surgical visualization platforms described herein can be used in combination with robotic surgical systems, but the surgical visualization platforms are not limited to use with robotic surgical systems. In certain examples, highly advanced surgical visualization can be performed without a robot and / or in a state where robotic assistance is limited and / or optional. Similarly, digital surgery can be performed without a robot and / or in a state where robotic assistance is limited and / or optional.

[0014] In certain examples, a surgical system incorporating the surgical visualization platform can enable smart incisions to identify and avoid important structures. Examples of important structures include anatomical structures, such as arteries like the ureter, superior mesenteric artery, veins like the portal vein, nerves like the phrenic nerve, and / or tumors, among other anatomical structures. In other examples, important structures can be foreign structures in an anatomical region, such as surgical devices, surgical fasteners, clips, clasps, sponges, bands, and / or plates. The important structures can be determined on a patient-by-patient and / or treatment-by-treatment basis. Exemplary important structures are further described herein. Smart incision techniques can provide improved intraoperative guidance for incisions and / or enable more intelligent decision-making, for example, by detection and avoidance techniques for important anatomical structures.

[0015] A surgical system incorporating a surgical visualization platform may also enable smart anastomosis that provides more consistent anastomosis at optimal location(s) by an improved workflow. Cancer localization methods may also be improved by the various surgical visualization platforms and procedures described herein. For example, cancer localization methods can identify and track the location, orientation, and margins of cancer. In certain examples, cancer localization methods can compensate for the movement of instruments, patients, and / or the patient's anatomical structures during surgery and provide guidance to return the clinician to the point of interest.

[0016] In certain aspects of the present disclosure, the surgical visualization platform may provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize the type and health of tissue without the need for physical tactile sensation, particularly at the time of incision and / or placement of a stapling device within the tissue. Certain tissue characterization techniques described herein can be utilized without using ionizing radiation and / or contrast agents. With regard to lymph node diagnosis and mapping, the surgical visualization platform can, for example, preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in cancer diagnosis and staging.

[0017] During surgery, the information available to the clinician via "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures, such as those embedded or buried within an organ, may be at least partially hidden from view, i.e., not visible. Additionally, certain dimensions and / or relative distances may be difficult to confirm with existing sensor systems and / or difficult to grasp with the "naked eye". Further, certain structures may move preoperatively (e.g., prior to surgery but after preoperative scans) and / or during surgery. In such examples, the clinician may not be able to accurately determine the location of important structures during surgery.

[0018] When the location of critical structures is uncertain and / or the proximity between critical structures and surgical instruments is unclear, it can impede the clinician's decision-making process. For example, a clinician can avoid a particular area to avoid inadvertent incision of critical structures. However, the avoided area may be unduly large and / or at least partially misplaced. Due to uncertainty and / or excessive caution, a clinician may not be able to reach a particular desired area. For example, even if the critical structure is not in that particular area and / or the clinician's actions in that particular area have no adverse effects, due to overcaution, the clinician may try to avoid the critical structure and leave behind a portion of the tumor and / or other undesirable tissue. In certain examples, surgical outcomes may improve as knowledge and / or certainty increase, which can enable the surgeon to be more accurate and, in certain examples, more restrained / aggressive with respect to a particular anatomical area.

[0019] In various aspects, the present disclosure provides a surgical visualization system for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide a high level of visualization capabilities beyond what a clinician can "see with the naked eye" and / or what an imaging system can recognize and / or communicate to the clinician. Various surgical visualization systems can improve outcomes in various examples by reinforcing and enhancing what a clinician can know prior to tissue treatment (e.g., incision).

[0020] For example, a visualization system can include a first light emitter configured to emit a plurality of spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, a molecular response to the spectral waves (spectroscopic imaging), and / or the light pattern. Through the following disclosure, it should be noted that unless specifically referring to visible light, all references to "light" can include electromagnetic waves or photons in the visible and / or non-visible portions of the electromagnetic radiation (EMR) wavelength spectrum. A surgical visualization system can also include an imaging system and a control circuit that signals communicate between the receiver(s) and the imaging system. Based on the output from the receiver(s), the control circuit can determine a geometric surface map of the visible surface at the surgical site, i.e., a three-dimensional surface topography, and one or more distances to the surgical site. In certain examples, the control circuit can determine one or more distances to at least partially hidden structures. Additionally, the imaging system can communicate the geometric surface map and the one or more distances to the clinician. In such examples, the enhanced view of the surgical site provided to the clinician can provide a display of hidden structures within the context related to the surgical site. For example, the imaging system can virtually enhance hidden structures on the geometric surface map of the tissue that is hiding and / or obscuring, similar to lines drawn on the ground to show utility pipes below the surface. Additionally, or alternatively, the imaging system can communicate the proximity of one or more surgical instruments to the visible obscured tissue and / or to at least partially hidden structures, and / or the depth of hidden structures below the visible surface of the obscuring tissue. For example, the visualization system can determine the distance to an extended line on the surface of the visible tissue and communicate that distance to the imaging system.

[0021] In various aspects of the present disclosure, a surgical visualization system for intraoperative identification and avoidance of critical structures is disclosed. Such a surgical visualization system can provide useful information to a clinician during a surgical procedure. As a result, while recognizing that the surgical visualization system is tracking critical structures, such as, for example, ureters, certain nerves, and / or important blood vessels that may be approached during an incision, the clinician can maintain momentum throughout the surgical procedure. In one aspect, the surgical visualization system can provide an indication to the clinician within a time sufficient for the clinician to pause and / or decelerate the surgical procedure and evaluate the proximity to the structure to prevent inadvertent damage to the critical structure. The surgical visualization system provides an ideal, optimized, and / or customizable amount of information to the clinician to avoid inadvertent damage to healthy tissue and / or critical structure(s) while enabling the clinician to move through the tissue securely and / or rapidly, thereby minimizing the risk of damage resulting from the surgical procedure.

[0022] FIG. 1 is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of important structures 101 within an anatomical region. The surgical visualization system 100 can be used, for example, in clinical analysis and / or medical intervention. In a particular example, the surgical visualization system 100 can be used during surgery to provide a clinician with real-time or near real-time information regarding proximity data, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured to facilitate intraoperative identification of the important structure(s) and / or to avoid the important structure(s) 101 by a surgical device. For example, by identifying the important structure 101, a clinician can avoid operating a surgical device around the important structure 101 and / or a predefined proximal region of the important structure 101 during a surgical procedure. The clinician can avoid, for example, incisions in, for example, veins, arteries, nerves, and / or blood vessels, which are identified as the important structure 101, and / or in the vicinity thereof. In various examples, the important structure 101 may be determined on a patient-by-patient and / or treatment-by-treatment basis.

[0023] The surgical visualization system 100 incorporates tissue identification and geometric surface mapping in combination with a distance sensor system 104. When combined, these features of the surgical visualization system 100 can determine the location of critical structures 101 within an anatomical region and / or the proximity of a surgical device 102 to the surface 105 of visible tissue and / or critical structures 101. Additionally, the surgical visualization system 100 includes an imaging system that includes an imaging device 120, such as a camera, configured to provide, for example, a real-time view of the surgical site. In various examples, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of an image based on molecular responses to various wavelengths. The view from the imaging device 120 can be provided to a clinician and, in various aspects of the present disclosure, enhanced with additional information based on tissue identification, landscape mapping, and the distance sensor system 104. In such examples, the surgical visualization system 100 includes a plurality of subsystems, namely, an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems can cooperate to provide highly synthesized data and integrated information to the clinician(s) during surgery.

[0024] The imaging device can include, for example, a camera or an imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In various aspects of the present disclosure, the imaging system can include an imaging device such as an endoscope. Additionally, or alternatively, the imaging system can include an imaging device such as, for example, an arthroscope, a vascular scope, a bronchoscope, a cholangioscope, a colonoscope, a cystoscope, a duodenoscope, an enteroscope, an esophagogastroduodenoscope (stomach camera), a laryngoscope, a nasopharyngo-ureteroscope, a sigmoidoscope, a thoracoscope, a ureteroscope, or an external scope. In other examples, such as for open surgery applications, the imaging system may not include a scope.

[0025] In various aspects of the present disclosure, the tissue-specific subsystem can be achieved using a spectral imaging system. The spectral imaging system can rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, titled "System and method for gross anatomic pathology using hyperspectral imaging," issued on March 1, 2016, which is hereby incorporated by reference in its entirety.

[0026] In various aspects of the present disclosure, the surface mapping subsystem can be achieved using an optical pattern system, as further described herein. The use of optical patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be utilized in the surgical visualization systems described herein.

[0027] Structured light is the process of projecting a known pattern (often a grid or horizontal bar) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, titled "SET COMPRISING A SURGICAL INSTRUMENT," published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, titled "DEPICTION SYSTEM," published on September 7, 2017, disclose surgical systems comprising a light source and a projector for projecting an optical pattern. U.S. Patent Application Publication No. 2017 / 0055819, titled "SET COMPRISING A SURGICAL INSTRUMENT," published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, titled "DEPICTION SYSTEM," published on September 7, 2017, are hereby incorporated by reference in their entireties.

[0028] In various aspects of the present disclosure, a distance determination system can be incorporated into a surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of a visible surface and determine various distances to the visible surface. Additionally, or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to specified tissue (or other structures) at a surgical site.

[0029] FIG. 2 is a schematic diagram of a control system 133 that can be utilized with a surgical visualization system 100. The control system 133 includes a control circuit 132 that communicates with a memory 134. The memory 134 stores instructions executable by the control circuit 132 for determining and / or recognizing critical structures (e.g., the critical structure 101 of FIG. 1), determining and / or calculating one or more distances and / or three-dimensional digital displays, and communicating specific information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logics 136, 138, 140, and 141. Also, the control system 133 includes an imaging system 142 having one or more cameras 144 (such as the imaging device 120 of FIG. 1), one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 can include one or more image sensors 135 (especially, for example, visible light, spectral imager, three-dimensional lens) for receiving signals from various light sources that emit light in various visible and invisible spectra. The display 146 can include one or more screens or monitors for depicting real, virtual, and / or virtually extended images and / or information to one or more clinicians.

[0030] In various aspects, the center of camera 144 is image sensor 135. Generally, state-of-the-art image sensors 135 are solid-state electronic devices that contain up to millions of individual light-detecting sites called pixels. The technology of image sensor 135 falls into one of two categories: charge-coupled device (CCD) imagers and complementary metal-oxide-semiconductor (CMOS) imagers. More recently, short-wave infrared (SWIR) has emerged as a new technology in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name “sCOMS”) and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 have sensitivity to wavelengths in the range of approximately 350 to 1050 nm, although this range is typically 400 to 1000 nm. CMOS sensors generally have higher sensitivity to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, are unable to distinguish colors. Thus, there are two types of color CCD cameras: one-chip and three-chip. One-chip color CCD cameras provide a commonly employed low-cost imaging solution. They use a mosaic (e.g., Bayer) optical filter to separate incident light into a series of colors and an interpolation algorithm to resolve a full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras provide higher resolution by employing a prism to direct each section of the incident spectrum to a different chip. Since each point in the object's space has separate RGB intensity values rather than using an algorithm to determine color, more accurate color reproduction is possible. Three-chip cameras provide very high resolution.

[0031] The control system 133 also includes a spectral light source 150 and a structured light source 152. In certain examples, a single light source can be pulsed to emit wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, a single light source can be pulsed to supply wavelengths of light within the visible spectrum (e.g., infrared spectral light) and wavelengths of light above the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various examples, the tissue identification logic 140 can identify the critical structure(s) via data from the spectral light source 150 received by the image sensor 135 portion of the camera 144. The surface mapping logic 136 can determine the contour of the surface of the visible tissue based on the reflected structured light. By time-of-flight measurement, the distance determination logic 141 can determine one or more distances to the visible tissue and / or the critical structure 101. One or more outputs from the surface mapping logic 136, the tissue identification logic 140, and the distance determination logic 141 can be provided to the imaging logic 138 and combined, integrated, and / or overlaid to be communicated to the clinician via the display 146 of the imaging system 142.

[0032] Next, FIGS. 2A to 2C will be briefly described, and various aspects of the control circuit 132 for controlling various aspects of the surgical visualization system 100 will be described. Referring to FIG. 2A, a control circuit 400 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure is illustrated. The control circuit 400 can be configured to implement various processes described herein. The control circuit 400 may include a microcontroller including one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions for implementing various processes described herein. The processor 402 may be any one of a number of single-core or multi-core processors known in the art. The memory circuit 404 can include volatile and non-volatile storage media. The processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0033] FIG. 2B shows a combinational logic circuit 410 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure. The combinational logic circuit 410 can be configured to implement various processes described herein. The combinational logic circuit 410 may include a finite state machine including combinational logic 412 configured to receive data associated with a surgical instrument or tool at an input 414, process the data by the combinational logic 412, and supply an output 416.

[0034] FIG. 2C shows a sequential logic circuit 420 configured to control aspects of a surgical visualization system 100 according to at least one aspect of the present disclosure. The sequential logic circuit 420 or combinational logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 may comprise a finite state machine. The sequential logic circuit 420 may comprise, for example, combinational logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store the current state of the finite state machine. In a particular example, the sequential logic circuit 420 may be synchronous or asynchronous. The combinational logic 422 is configured to receive data associated with a surgical device or system from input 426, process the data by the combinational logic 422, and supply an output 428. In other aspects, the circuit may comprise a combination of a processor (e.g., processor 402 of FIG. 2A) and a finite state machine implementing various processes herein. In other aspects, the finite state machine may comprise a combination of a combinational logic circuit (e.g., combinational logic circuit 410 of FIG. 2B) and the sequential logic circuit 420.

[0035] Referring again to the surgical visualization system 100 of FIG. 1, the critical structure 101 can be an anatomical structure of interest. For example, the critical structure 101 can be, among other anatomical structures, arteries such as the ureter, superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors. In other examples, the critical structure 101 can be a foreign object structure in an anatomical region, such as a surgical device, surgical fastener, clip, retainer, bovie, band, and / or plate. Exemplary critical structures are further described in the foregoing U.S. patent applications, including this specification and, for example, U.S. Patent Application No. 16 / 128,192, filed September 11, 2018, entitled "VISUALIZATION OF SURGICAL DEVICES", each of which is hereby incorporated by reference in its entirety.

[0036] In one aspect, the critical structure 101 can be embedded in the tissue 103. In other words, the critical structure 101 can be disposed below the surface 105 of the tissue 103. In such an example, the tissue 103 hides the critical structure 101 from the clinician's view. The critical structure 101 is also shielded from the view of the imaging device 120 by the tissue 103. The tissue 103 can be, for example, fat, connective tissue, adhesions, and / or an organ. In other examples, the critical structure 101 can be partially shielded from view.

[0037] FIG. 1 also shows a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from the distal end of the shaft of the surgical device 102. The surgical device 102 can be any suitable surgical device, such as, for example, a cutting instrument, a stapler, a grasping instrument, a clip applier, and / or an energy device including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector. Additionally, or alternatively, the surgical device 102 can include another imaging modality or diagnostic modality, such as, for example, an ultrasonic device. In one aspect of the present disclosure, the surgical visualization system 100 can be configured to achieve identification of one or more critical structures 101 and access of the surgical device 102 to the critical structure(s) 101.

[0038] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include a plurality of lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 can be a hyperspectral camera, a multispectral camera, or a selective spectral camera, as further described herein. The imaging device 120 can also include a waveform sensor 122 (e.g., a spectral image sensor, a detector, and / or a three-dimensional camera lens). For example, the imaging device 120 can be used together to generate a three-dimensional image of the surgical site, render the three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site by simultaneously recording two two-dimensional images. The imaging device 120 can include a right lens and a left lens. Additionally or alternatively, the imaging device 120 can be configured to receive images showing the topography of visible tissue and the identification and location of hidden critical structures, as further described herein. For example, as shown in FIG. 1, the field of view of the imaging device 120 can be overlapped with a light pattern (structured light) on the surface 105 of the tissue.

[0039] In one aspect, the surgical visualization system 100 may be incorporated into the robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include rigid structural members 116 and joints 118 that can include servo motor control. The first robotic arm 112 is configured to operate the surgical device 102, and the second robotic arm 114 is configured to operate the imaging device 120. The robotic control unit can be configured to generate control movements to the robotic arms 112, 114 that can act on, for example, the surgical device 102 and the imaging device 120.

[0040] The surgical visualization system 100 also includes an emitter 106 configured to emit light patterns such as stripes, grid lines, and / or dots to enable determination of the topography or landscape of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from an emitter 106 located, for example, on one of the surgical device 102 and / or the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is employed to determine the surface 105 of the tissue 103 and / or the shape defined during surgery by the movement of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances to the surface 105.

[0041] In one aspect, the imaging device 120 may also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 and reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effect of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted from the optical waveform emitter 123 can be configured to enable the identification of anatomical and / or body structure types such as the critical structure 101. The identification of the critical structure 101 can be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 can include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other examples, the waveform sensor 122 and the optical waveform emitter 123 can include, for example, a photoacoustic imaging system. In other examples, the optical waveform emitter 123 can be disposed on a surgical device separate from the imaging device 120.

[0042] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one aspect, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system that includes an emitter such as emitter 106 and a receiver 108 that may be disposed on the surgical device 102. In other examples, the time-of-flight emitter may be separate from the structured light emitter. In one general aspect, the emitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a coincidence sensor. The time-of-flight distance sensor system 104 can detect the "time of flight", that is, the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. By using a very narrow light source in the emitter 106, the distance sensor system 104 is enabled to determine the distance to the surface 105 of the tissue 103 immediately in front of the distance sensor system 104. Referring further to FIG. 1, d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue. The distance sensor system 104 is employed to determine the emitter-tissue distance d e . The device-tissue distance d t can be obtained from the known position of the emitter 106 on the shaft of the surgical device 102 relative to the distal end of the surgical device 102. In other words, if the distance between the emitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t can be determined from the emitter-tissue distance d e . In a particular example, the shaft of the surgical device 102 can include one or more articulating joints and can be articulable with respect to the emitter 106 and the joe. The articulating configuration can include, for example, a multi-articulating vertebral-like structure. In a particular example, a three-dimensional camera can be utilized to triangulate one or more distances to the surface 105.

[0043] In various examples, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on a separate surgical device instead of the surgical device 102. For example, the receiver 108 can be mounted on a cannula or trocar that extends therethrough such that the surgical device 102 reaches the surgical site. In yet other examples, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on another robotic control arm (e.g., robotic arm 114), on another robotically controlled movable arm, and / or on an operating room (OR) table or fixture. In a particular example, the imaging device 120 includes a time-of-flight receiver 108 that determines the distance from the emitter 106 to the surface 105 of the tissue 103 using the line between the emitter 106 on the surgical device 102 and the imaging device 120. For example, based on the known positions of the emitter 106 of the time-of-flight distance sensor system 104 (on the surgical device 102) and the receiver 108 (on the imaging device 120), the distance d e can be triangulated. The three-dimensional position of the receiver 108 can be known and / or aligned relative to the robotic coordinate plane during surgery.

[0044] In a particular example, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other examples, the surgical visualization system 100 can be utilized separately from the robotic system. In such examples, the distance sensor system 104 may be independent of the robotic system.

[0045] In certain examples, one or more of the robotic arms 112, 114 may be separate from the main robotic system used in surgery. At least one of the robotic arms 112, 114 can be placed and aligned in a particular coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 can control and / or align the position of the robotic arm(s) 112, 114 relative to a particular coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 can be aligned relative to a particular coordinate system.

[0046] Referring further to FIG. 1, d w is the camera-important structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the important structure 101, and d A is the depth of the important structure 101 below the surface 105 of the tissue 103 (i.e., the distance between a portion of the surface 105 closest to the surgical device 102 and the important structure 101). In various aspects, the time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-important structure distance d w . The use of spectral imaging in combination with a time of flight sensor is further described herein. Further, referring now to FIG. 3, in various aspects of the present disclosure, the depth d A of the important structure 101 relative to the surface 105 of the tissue 103 is the distance d w , as well as the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and thus the distance d x ) and can be determined by triangulation from the sum of the distance d e and d A to obtain the distance d y .

[0047] Additionally, or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or waveform range) can be utilized to determine the camera-critical structure distance d w and a second waveform (or waveform range) can be utilized to determine the distance to the surface 105 of the tissue 103. In such an example, different waveforms can be utilized to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0048] Additionally, or alternatively, in certain examples, the distance d A can be determined from ultrasound, a registered magnetic resonance view (MRI), or a computed tomography (CT) scan. In still other examples, the distance d A can be determined by spectral imaging since the detection signal received by the imaging device can vary based on the type of material. For example, fat can decrease the detection signal in a first way, or by a first amount, and collagen can decrease the detection signal in a different second way, or by a second amount.

[0049] Referring now to the surgical visualization system 160 of FIG. 4, the surgical device 162 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect the reflected waveform. The optical waveform emitter 123 can be configured to emit waveforms for determining the distances d t and d w from a common device such as the surgical device 162, as further described herein. In such an example, the distance d A from the surface 105 of the tissue 103 to the surface of the critical structure 101 can be determined as follows. d A = d w - d t .

[0050] As disclosed herein, various information regarding visible tissue, embedded critical structures, and surgical devices can be determined by utilizing an approach that incorporates one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays in combination with an image sensor configured to detect spectral wavelengths and structured light arrays. Further, the image sensor can be configured to provide an image of the surgical site to the imaging system by receiving visible light. Logic or algorithms are employed to identify information received from the time-of-flight sensors, spectral wavelengths, structured light, and visible light and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various examples, the imaging device 120 can include a plurality of image sensors.

[0051] Camera - Critical Structure Distance d w It can also be detected by one or more alternative methods. In one aspect, for example, fluorescence fluoroscopy visualization techniques such as indocyanine green (ICG) can be utilized to illuminate critical structures 201 as shown in FIGS. 6 - 8. The camera 220 can include two optical waveform sensors 222, 224 that simultaneously capture left and right images of the critical structure 201 (FIGS. 7A and 7B). In such an example, the camera 220 can depict the emission of the critical structure 201 below the surface 205 of the tissue 203, and the distance d w can be determined by the known distance between sensors 222 and 224. In certain examples, the distance can be determined more accurately by utilizing two or more cameras or by moving the camera between multiple positions. In a particular aspect, one camera can be controlled by a first robotic arm and a second camera can be controlled by another robotic arm. In such a robotic system, one camera can be, for example, a passive camera on a passive arm. The passive arm and the camera thereon can be programmed, for example, to track another camera and maintain a specific distance and / or lens angle.

[0052] In yet another aspect, the surgical visualization system 100 may employ two separate waveform receivers (i.e., cameras / image sensors) to determine d w Referring now to FIG. 9, where the critical structure 301 or its contents (e.g., a blood vessel or the contents of a blood vessel) may emit a signal 302, for example, by fluoroscopy, the actual position can be triangulated from two separate cameras 320a, 320b at known positions.

[0053] Referring now to FIGS. 10A and 10B, in another aspect, the surgical visualization system may employ a dithering or moving camera 440 to determine the distance d w The camera 440 is robotically controlled such that the three-dimensional coordinates at different positions of the camera 440 are known. In various examples, the camera 440 can pivot at a cannula or patient interface. For example, where the critical structure 401 or its contents (e.g., a blood vessel or the contents of a vessel) may emit a signal, for example, by fluoroscopy, the actual position can be triangulated from a camera 440 that rapidly moves between two or more known positions. In FIG. 10A, the camera 440 is moved axially along axis A. More specifically, the camera 440 is translated along axis A to a position shown as position 440' closer to the critical structure 401, for example, by moving in and out with a robotic arm, by a distance d 1 As the camera 440 moves the distance d 1 and the size of the view changes with respect to the critical structure 401, the distance to the critical structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d 1 ) may correspond to an angle θ of 6.28 degrees 1 and an angle θ of 8.19 degrees 2 . Additionally, or alternatively, the camera 440 can rotate or sweep along an arc between different positions. Referring now to FIG. 10B, the camera 440 moves axially along axis A and rotates about axis A by θ 3Rotates. The pivot point 442 for the camera 440 to rotate is disposed on the cannula / patient interface. In FIG. 10B, the camera 440 translates and rotates to position 440''. As the camera 440 moves and the edge of the view changes with respect to the critical structure 401, the distance to the critical structure 401 can be calculated. In FIG. 10B, the distance d 2 can be, for example, 9.01 mm, and the angle θ 3 can be, for example, 0.9 degrees.

[0054] FIG. 5 shows a surgical visualization system 500 that is similar to the surgical visualization system 100 in many respects. In various examples, the surgical visualization system 500 can be a further illustration of the surgical visualization system 100. Similar to the surgical visualization system 100, the surgical visualization system 500 includes a surgical device 502 and an imaging device 520. The imaging device 520 includes, for example, a spectral light emitter 523 configured to emit spectral light of a plurality of wavelengths to obtain a spectral image of hidden structures. The imaging device 520 can also include, in various examples, a three-dimensional camera and associated electronic processing circuitry. Illustrated is a surgical visualization system 500 that is utilized during surgery to identify specific critical structures such as the ureter 501a and blood vessels 501b within an organ 503 (in this example, the uterus) that are not visible on the surface and to facilitate avoiding them.

[0055] The surgical visualization system 500 is configured to determine the emitter-tissue distance d e from the emitter 506 on the surgical device 502 to the surface 505 of the uterus 503 by structured light. The surgical visualization system 500 is configured to extrapolate the device-tissue distance d e from the surgical device 502 to the surface 505 of the uterus 503 based on the emitter-tissue distance d t . Also, the surgical visualization system 500 determines the tissue-ureter distance d A from the ureter 501a to the surface 505 and the camera-ureter distance d wconfigured to determine. As described herein with respect to FIG. 1, for example, surgical visualization system 500 can determine, e.g., by spectral imaging and time-of-flight sensors, distance d w In various examples, surgical visualization system 500 can determine (e.g., triangulate) tissue-ureter distance d A (i.e., depth) based on other distances and / or surface mapping logic described herein.

[0056] Referring now to FIG. 11, a schematic diagram of a control system 600 for a surgical visualization system, such as surgical visualization system 100, is shown. Control system 600 is, in particular, a conversion system that integrates tissue identification by spectral signature and tissue positioning by structured light to identify these structures, particularly when the critical structures are obscured by other tissues, such as, for example, fat, connective tissue, blood, and / or other organs. Such techniques can also be useful for detecting tissue anomalies, such as differentiating healthy tissue within an organ from tumors and / or diseased tissue.

[0057] The control system 600 is configured to implement a hyperspectral imaging visualization system, in which molecular responses are utilized to detect and identify anatomical structures within the surgical field. The control system 600 includes a conversion logic circuit 648 for converting tissue data into information usable by a surgeon. For example, variable reflectance based on wavelengths for shielding materials can be utilized to identify critical structures within anatomical structures. Further, the control system 600 combines the identified spectral signature and structured light data within an image. For example, the control system 600 can be employed to create a three-dimensional dataset for surgical applications in a system using an extended image overlay. The technology can employ additional visual information for use both during and prior to surgery. In various examples, the control system 600 is configured to provide a warning to a clinician upon proximity to one or more critical structures. Various algorithms can be employed to guide robotic automation and semi-automation approaches based on the surgical procedure and proximity to the critical structure(s).

[0058] Employ a projected light array to determine the shape and movement of tissue during surgery. Alternatively, a flash lidar may be utilized for surface mapping of tissue.

[0059] The control system 600 is configured to detect critical structure(s), provide an image overlay of the critical structure, and measure the distance to the surface of visible tissue and the distance to the embedded / concealed critical structure(s). In other examples, the control system 600 can measure the distance to the surface of visible tissue or detect critical structure(s) and provide an image overlay of the critical structure.

[0060] The control system 600 includes a spectrum control circuit 602. The spectrum control circuit 602 can be, for example, a field programmable gate array (FPGA) or another suitable circuit configuration as described herein in connection with FIGS. 2A-2C. The spectrum control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectrum processing and can utilize, for example, C / C++ code. The video input processor 606 receives video input terminals of control (metadata) data such as, for example, shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608 that includes hyperspectrum video output terminals for interface control (metadata) data. The video output processor 608 provides the video output signal to an image overlay controller 610.

[0061] The video input processor 606 is connected to a patient-side camera 612 via a patient isolation circuit 614. As described above, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit can include a plurality of transformers so that the patient is isolated from other circuits within the system. The camera 612 receives intraoperative images via an optical element 632 and the image sensor 634. The image sensor 634 can include, for example, a CMOS image sensor or can include any of the image sensor technologies discussed herein in connection with FIG. 2. In one aspect, the camera 612 outputs an image with a 14-bit / pixel signal. It will also be understood that higher or lower pixel resolutions can be employed without departing from the scope of the present disclosure. The separated camera output signal 613 is provided to a color RGB fusion circuit 616 that employs a hardware register 618 and a Nios2 coprocessor 620 to process the camera output signal 613. A color RGB fusion output signal is provided to the video input processor 606 and a laser pulse control circuit 622.

[0062] The laser pulse control circuit 622 controls the laser light engine 624. The laser light engine 624 outputs light at a plurality of wavelengths (λ 1 , λ 2 , λ 3 ... λ n ) including near-infrared (NIR). The laser light engine 624 can operate in multiple modes. In one aspect, the laser light engine 624 can operate in, for example, two modes. In the first mode, for example, the standard operating mode, the laser light engine 624 outputs an illumination signal. In the second mode, for example, the specific mode, the laser light engine 624 outputs RGBG light and NIR light. In various examples, the laser light engine 624 can operate in a polarization mode.

[0063] The light output 626 from the laser light engine 624 illuminates the targeted anatomical structure within the surgical site 627 during surgery. The laser pulse control circuit 622 also controls a laser pulse controller 628 for a laser pattern projector 630 that projects a laser light pattern 631, such as a grid or pattern of lines and / or dots, at a predetermined wavelength (λ 2 ) onto the surgical tissue or organ at the surgical site 627. The camera 612 receives the patterned light and the reflected light output through the camera optical element 632. The image sensor 634 converts the received light into a digital signal.

[0064] The color RGB fusion circuit 616 also outputs a signal to the image overlay controller 610 and a video input module 636 for reading the laser light pattern 631 projected onto the targeted anatomical structure of the surgical site 627 by the laser pattern projector 630. The processing module 638 processes the laser light pattern 631 and outputs a first video output signal 640 representing the distance to the visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the targeted anatomical structure at the surgical site.

[0065] The first video output signal 640 and the second video output signal 642 include data representing the positions of important structures on the three-dimensional surface model provided to the integrated module 643. In combination with the data from the video output processor 608 of the spectral control circuit 602, the integrated module 643 determines the distance d to the buried important structure A (FIG. 1), for example, by a triangulation algorithm 644, and the distance d A can be provided to the image overlay controller 610 via the video output processor 646. The aforementioned conversion logic can include a conversion logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 disposed at the surgical site 627.

[0066] Preoperative data 650 by CT or MRI scan can be employed to align or register specific three-dimensional deformable tissues in various examples. Such preoperative data 650 can be provided to the integrated module 643 and ultimately to the image overlay controller 610, so that such information can be overlaid with the view of the camera 612 and provided to the video monitor 652. The alignment of the preoperative data is further described in this specification and in the aforementioned U.S. patent applications, including, for example, U.S. Patent Application No. 16 / 128,195, filed September 11, 2018, entitled "INTEGRATION OF IMAGING DATA", each of which is hereby incorporated by reference in its entirety.

[0067] The video monitor 652 can output an integrated / extended view from the image overlay controller 610. A clinician can select and / or switch between different views on one or more monitors. On the first monitor 652a, the clinician can switch between (A) a view in which a three-dimensional rendering of the visible tissue is shown and (B) an extended view in which one or more hidden critical structures are depicted over the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can switch, for example, between one or more hidden critical structures and / or distance measurements to the surface of the visible tissue.

[0068] The control system 600, and / or its various control circuits, can be incorporated into the various surgical visualization systems disclosed herein.

[0069] FIG. 12 shows a structured (or patterned) light system 700 according to at least one aspect of the present disclosure. As described herein, structured light, for example in the form of stripes or lines, is projected from a light source and / or projector 706 onto the surface 705 of a targeted anatomical structure to identify the shape and contour of the surface 705. For example, a camera 720, which may be similar in various respects to the imaging device 120 (FIG. 1), can be configured to detect the projected light pattern on the surface 705. When the projected pattern deforms upon striking the surface 705, it becomes possible for the vision system to calculate the depth and surface information of the targeted anatomical structure.

[0070] In certain examples, invisible (i.e., undetectable) structured light can be utilized, and the structured light is utilized without interfering with other computer vision tasks where the projected pattern may be confused. For example, infrared light or visible light at a very fast frame rate that repeats two exactly opposite patterns can be used to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0071] As described above, using the various surgical visualization systems described herein, it is possible to visualize various different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that may be prevented from being visualized by EMR in the visible portion of the spectrum. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize various types of tissues based on various combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the visualized tissue based on the absorption coefficients of the tissue over various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the visualized tissue based on a particular combination of constituent materials. By way of example, FIG. 13A is a graph 2300 showing how the absorption coefficients of various biological materials vary over the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 represents the absorption coefficient of the biological material (e.g., cm -1 units), and the horizontal axis 2304 represents the EMR wavelength (e.g., in μm units). This graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Since different tissue types have different combinations of constituent materials, the tissue type(s) visualized by the surgical visualization system can be identified and differentiated according to the particular combination of detected constituent materials. Thus, the spectral imaging system can be configured to emit EMR at many different wavelengths, determine the constituent materials of the tissue based on the absorption EMR absorption responses detected at different wavelengths, and then characterize the tissue type based on the particular detected combination of constituent materials.

[0072] An example of the use of spectral imaging techniques to visualize various tissue types and / or anatomical structures is shown in FIG. 13B. In FIG. 13B, a spectral emitter 2320 (e.g., spectral light source 150) is utilized by an imaging system to visualize a surgical site 2325. The EMR emitted by the spectral emitter 2320 and reflected from the tissue and / or structures of the surgical site 2325 is received by an image sensor 135 (FIG. 2), and the tissue and / or structures can be visualized, and this visualization can be either visible (e.g., located on the surface of the surgical site 2325) or obscured (e.g., under other tissue and / or structures at the surgical site 2325). In this example, the imaging system 142 (FIG. 2) is based on spectral signatures characterized by different absorption characteristics (e.g., absorption coefficients) of the constituent materials for each of the various tissue / structure types, and can visualize a tumor 2332, an artery 2334, and various abnormalities 2338 (i.e., tissues that cannot be identified relative to known or expected spectral signatures). The visualized tissue and structures can be displayed on a display screen associated with or coupled to the imaging system 142, such as the imaging system display 146 (FIG. 2), the primary display 2119 (FIG. 18), the non-sterile display 2109 (FIG. 18), the hub display 2215 (FIG. 19), the device / instrument display 2237 (FIG. 19), etc.

[0073] Furthermore, the imaging system 142 can be configured to adjust or update the visualization of the displayed surgical site according to the identified tissue and / or structure type. For example, the imaging system 142 can display a margin 2330a associated with a tumor 2332 visualized on a display screen (e.g., display 146). The margin 2330a can indicate the area or amount of tissue to be resected to ensure complete removal of the tumor 2332. The control system 133 (FIG. 2) can be configured to control or update the dimensions of the margin 2330a based on the tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 has identified a plurality of abnormalities 2338 within the FOV. Accordingly, the control system 133 can adjust the displayed margin 2330a to a first updated margin 2330b having dimensions sufficient to encompass this abnormality 2338. Also, the imaging system 142 has identified an artery 2334 that partially overlaps the initially displayed margin 2330a (shown by the highlighted region 2336 of the artery 2334). Accordingly, the control system 133 can adjust the displayed margin 2330a to a second updated margin 2330c having dimensions sufficient to encompass the relevant portion of this artery 2334.

[0074] In addition to, or instead of, the absorption characteristics described above with respect to FIGS. 13A and 13B, the tissue and / or structure can be imaged or characterized according to the reflection characteristics of these tissues and structures across the EMR wavelength spectrum. For example, FIGS. 13C-13E show various graphs of the reflectance of different types of tissue or structure across various EMR wavelengths. FIG. 13C is an exemplary graphical representation 1050 of ureter signature vs. shield. FIG. 13D is an exemplary graphical representation 1052 of artery signature vs. shield. FIG. 13E is an exemplary graphical representation 1054 of nerve signature vs. shield. The plots in FIGS. 13C-13E represent the reflectance as a function of the wavelength (in nm) of a particular structure (ureter, artery, and nerve) relative to the corresponding reflectance of fat, lung tissue, and blood at the corresponding wavelength. These graphs are for illustrative purposes only, and it should be understood that other tissues and / or structures may include corresponding detectable reflectance signatures that enable the identification and visualization of the tissue and / or structure.

[0075] In various examples, the selected wavelengths for spectral imaging can be identified and utilized based on the critical structures and / or shields predicted at the surgical site (i.e., "selective spectrum" imaging). By utilizing selective spectrum imaging, the amount of time required to acquire a spectral image can be minimized such that information can be acquired in real-time or near real-time and utilized during the surgery. In various examples, the wavelengths can be selected by a clinician or by a control circuit based on clinician input. In a particular example, the wavelengths can be selected based on, for example, machine learning and / or big data accessible to the control circuit via the cloud.

[0076] The foregoing examples of the application of spectral imaging to tissue can be utilized during surgery to measure the distance between a waveform emitter and an important structure obscured by the tissue. In one aspect of the present disclosure, referring now to FIGS. 14 and 15, a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125 is shown. The time-of-flight sensor system 1104 can be incorporated, in a particular example, into the surgical visualization system 100 (FIG. 1). The time-of-flight sensor system 1104 includes a waveform emitter 1106 and a waveform receiver 1108 on the same surgical device 1102. The emitted wave 1124 extends from the emitter 1106 to the important structure 1101, and the received wave 1125 is reflected from the important structure 1101 to the receiver 1108. The surgical device 1102 is disposed through a trocar 1110 that extends into the patient's cavity 1107.

[0077] The waveforms 1124, 1125 are configured to penetrate the obscuring tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 can be wavelengths in the NIR spectrum or the SWIR spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or a photoacoustic signal can be emitted from the emitter 1106 and can penetrate the tissue 1103 obscuring the important structure 1101. The emitted waveform 1124 can be reflected by the important structure 1101. The received waveform 1125 may be delayed due to the distance d between the distal end of the surgical device 1102 and the important structure 1101. In various examples, the waveforms 1124, 1125 can be selected to target the important structure 1101 within the tissue 1103 based on the spectral signature of the important structure 1101, as further described herein. In various examples, the emitter 1106 is configured to supply, for example, a binary signal representing on and off, as shown in FIG. 15, and can be measured by the receiver 1108.

[0078] Based on the delay between the emitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d (FIG. 14). The time-of-flight timing diagram 1130 of the emitter 1106 and the receiver 1108 in FIG. 14 is shown in FIG. 15. The delay is a function of the distance of the distance d, and the distance d is given as follows:

[0079]

Equation

[0080] As provided herein, the time of flight of the waveforms 1124, 1125 corresponds to the distance d in FIG. 14. In various examples, additional emitters / receivers and / or pulse signals from the emitter 1106 can be configured to emit non-transmissive signals. The non-transmissive tissue can be configured to determine the distance to the surface 1105 of the tissue 1103 that shields from the emitter. In various examples, the depth of the important structure 1101 can be determined by the following equation. d A = d w - d t . wherein, d A = the depth of the important structure 1101, d w = the distance from the emitter 1106 to the important structure 1101 (d in FIG. 14), and, d t = the distance from the emitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the shielding r tissue 1103.

[0081] In one aspect of the present disclosure, referring now to FIG. 16, a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c is shown. The time-of-flight sensor system 1204 can be incorporated into the surgical visualization system 100 (FIG. 1) in a particular example. The time-of-flight sensor system 1204 includes a waveform emitter 1206 and a waveform receiver 1208. The waveform emitter 1206 is disposed on the first surgical device 1202a, and the waveform receiver 1208 is disposed on the second surgical device 1202b. The surgical devices 1202a, 1202b are disposed through trocars 1210a, 1210b extending respectively within the patient's cavity 1207. The emitted waves 1224a, 1224b, 1224c extend from the emitter 1206 towards the surgical site, and the received waves 1225a, 1225b, 1225c are reflected from the receiver 1208 from various structures and / or surfaces at the surgical site.

[0082] The different emitted waves 1224a, 1224b, 1224c are configured to target different types of materials at the surgical site. For example, wave 1224a targets the shielding tissue 1203, wave 1224b targets the first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets the second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, 1224c may be wavelengths in the visible light, NIR, or SWIR spectra. For example, visible light can reflect off the surface 1205 of the tissue 1203, and NIR waveforms and / or SWIR waveforms can be configured to penetrate the surface 1205 of the tissue 1203. In various aspects, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or a photoacoustic signal can be emitted from the emitter 1206 as described herein. In various examples, waves 1224b, 1224c can be selected to target the critical structures 1201a, 1201b within the tissue 1203 based on the spectral signatures of the critical structures 1201a, 1201b as further described herein. Photoacoustic imaging is further described in various U.S. patent applications, each of which is incorporated herein by reference in its entirety.

[0083] The emitted waves 1224a, 1224b, 1224c can be reflected from the targeted materials (i.e., the surface 1205, the first critical structure 1201a, and the second structure 1201b, respectively). The received waveforms 1225a, 1225b, 1225c are delayed due to the distances d 1a , d 2a , d 3a , d 1b , d 2b , d 2c shown in FIG. 16.

[0084] In a time-of-flight sensor system 1204 in which the emitter 1206 and the receiver 1208 can be individually positioned (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), the various distances d 1a , d 2a , d 3a, d 1b , d 2b , d 2c can be calculated from the known positions of emitter 1206 and receiver 1208. For example, when surgical devices 1202a, 1202b are robotically controlled, the positions can be known. With knowledge of the positions of emitter 1206 and receiver 1208, as well as the time of the photon stream targeting a specific tissue and the information of that specific response received by receiver 1208, distance d 1a , d 2a , d 3a , d 1b , d 2b , d 2c , d

[0085] Referring further to FIG. 16, in various examples, in the view provided to the clinician, the center of mass of the target structure in the resulting image remains constant, i.e., within a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b, the receiver 1208 can be rotated. Such an orientation can quickly communicate one or more relevant distances and / or viewpoints with respect to the critical structure. For example, as shown in FIG. 16, the surgical site is displayed from a viewpoint where critical structure 1201a is perpendicular to the view plane (i.e., the blood vessel is facing in and out of the page). In various examples, such an orientation can be the default setting. However, the view can be rotated or adjusted by the clinician. In certain examples, the clinician can switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0086] In various examples, the receiver 1208 can be mounted on a trocar or cannula, such as trocar 1210b, through which the surgical device 1202b is disposed. In other examples, the receiver 1208 can be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 1208 can be attached to a movable arm separate from the robot that controls the surgical device 1202a, or can be mounted on an operating room (OR) table that can be aligned with the intraoperative robot coordinate plane. In such examples, the positions of the emitter 1206 and the receiver 1208 can be aligned in the same coordinate plane so that the distance can be triangulated from the output of the time-of-flight sensor system 1204.

[0087] A combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS) called TOF-NIRS, which is capable of measuring the time-resolved profile of NIR light with nanosecond resolution, can be found in the literature "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT" in Journal of the American Society for Horticultural Science, May 2013 vol.138 no.3 225~228, which is hereby incorporated by reference in its entirety and can be viewed at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0088] In various examples, the time-of-flight spectral waveform is configured to determine the depth of important structures and / or the proximity of the surgical device to the important structures. Further, the various surgical visualization systems disclosed herein include surface mapping logic configured to create a three-dimensional rendering on the surface of the visible tissue. In such examples, even if the important structures are blocked by the visible tissue, the clinician can recognize the proximity (or lack thereof) of the surgical device to the important structures. In one example, the topography of the surgical site is provided on a monitor by the surface mapping logic. When the important structures are close to the surface of the tissue, spectral imaging can communicate the location of the important structures to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other examples, spectral imaging can detect structures 10 or 20 mm below the surface of the tissue. Based on the known limitations of the spectral imaging system, the system is configured to communicate that the important structures are out of range if they are not simply detected by the spectral imaging system. Thus, the clinician can continue to move the surgical device and / or continue to manipulate the tissue. When the important structures move within the range of the spectral imaging system, the system can identify the structures and thus communicate that the structures are within range. In such examples, an alert can be provided when the structures are first identified and / or further moved within a predefined proximity zone. In such examples, proximity information (i.e., not in proximity) can be provided to the clinician even if the important structures are not identified by the spectral imaging system having a known boundary / range.

[0089] The various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of important structure(s) during surgery and to alert the clinician before an important structure(s) is damaged by an inadvertent incision and / or transection. In various aspects, the surgical visualization system is configured to identify one or more of the following important structures, e.g., ureters, intestines, rectum, nerves (including phrenic nerve, recurrent laryngeal nerve [RLN], facial nerve branches, vagus nerve, and their branches), blood vessels (including pulmonary and lobar arteries and veins, inferior mesenteric artery [IMA] and its branches, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its branches (including middle colic artery, right colic artery, ileocolic artery), hepatic artery and its branches, portal vein and its branches, splenic artery / vein and its branches, external and internal iliac vessels (lower abdomen), short gastric artery, uterine artery, median sacral vessels, and lymph nodes. Further, the surgical visualization system is configured to indicate the proximity of the surgical device(s) to the important structure(s) and / or to alert the clinician when the surgical device(s) approaches an important structure.

[0090] Various aspects of the present disclosure provide for identification of critical structures during surgery (e.g., identification of ureters, nerves, and / or blood vessels) and instrument approach monitoring. For example, various surgical visualization systems disclosed herein can include spectral imaging and surgical instrument tracking that enable visualization of critical structures below the surface of tissue, such as 1.0 to 1.5 cm below the surface of the tissue. In other examples, the surgical visualization system can identify structures less than 1.0 cm or greater than 1.5 cm below the surface of the tissue. For example, a surgical visualization system that can only identify structures within 0.2 mm of the surface can be beneficial, for example, when they would otherwise be invisible due to depth. In various aspects, the surgical visualization system can extend the clinician's view by, for example, virtually displaying critical structures as a visible white light image overlay on the surface of the visible tissue. The surgical visualization system can provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and can provide an approach alert when the distal tip of the surgical instrument moves within a defined range of a critical structure, such as within 1.0 cm of the critical structure.

[0091] The various surgical visualization systems disclosed herein can identify when an incision is too close to a critical structure. Based on temperature (i.e., being too hot near a critical structure where there is a risk of damaging / heating / melting the critical structure), and / or tension (i.e., being under too much tension near a critical structure where there is a risk of damaging / lacerating / pulling the critical structure), an incision can be "too close" to a critical structure. Such surgical visualization systems can, for example, facilitate incisions around blood vessels when skeletonizing the tissue around the vessels prior to ligation. In various examples, an infrared camera can be utilized to read the heat of the surgical site and provide warnings to the clinician based on the detected heat and the distance from the instrument to the structure. For example, an alert can be provided to the clinician at a first distance (e.g., 10 mm, etc.) when the temperature of the instrument exceeds a predetermined threshold (e.g., 120°F, etc.), and an alert can be provided to the clinician at a second distance (e.g., 5 mm, etc.) when the instrument temperature is below the predetermined threshold. The predetermined threshold and / or warning distance can be programmably set by default and / or by the clinician. Additionally, or alternatively, the proximity alert can be linked to a heat measurement performed by the instrument itself, such as a monopolar or bipolar cutting instrument, or a thermocouple that measures the heat within the distal jaw of a vessel sealer.

[0092] The various surgical visualization systems disclosed herein can provide sufficient sensitivity and specificity to a critical structure to enable a clinician to proceed with a rapid yet safe incision with confidence based on the standard of care and / or device safety data. This system can function in real-time during surgery with a minimal risk of ionizing radiation to the patient or clinician, and in various examples, there is no risk of ionizing radiation to the patient or clinician. In contrast, in a fluoroscopic procedure, the patient and clinician(s) can be exposed to ionizing radiation via, for example, an X-ray beam utilized to view anatomical structures in real-time.

[0093] The various surgical visualization systems disclosed herein can be configured to detect and identify one or more desired types of critical structures within the forward path of a surgical device, such as when the path of the surgical device is robotically controlled. Additionally, or alternatively, the surgical visualization system can be configured to detect and identify one or more types of critical structures, for example, in the surrounding area of the surgical device and / or in multiple planes / dimensions.

[0094] The various surgical visualization systems disclosed herein can be easily operated and / or interpreted. Further, the various surgical visualization systems can incorporate an "override" function that allows a clinician to override default settings and / or operations. For example, a clinician can selectively turn off alerts from the surgical visualization system and / or approach a critical structure closer than suggested by the surgical visualization system when the risk to the critical structure is lower than the risk of avoiding the area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue can be greater than the risk of damaging the critical structure).

[0095] The various surgical visualization systems disclosed herein can be incorporated into a surgical system and / or used during a surgical procedure with a limited impact on the workflow. In other words, the implementation of the surgical visualization system need not change the way the surgical procedure is performed. Further, the surgical visualization system can be economical compared to the cost of an inadvertent incision. The data shows a reduction in inadvertent damage to critical structures and can facilitate an increase in reimbursement amounts.

[0096] The various surgical visualization systems disclosed herein can operate in real-time, or nearly real-time, and well in advance to enable a clinician to predict critical structure(s). For example, the surgical visualization system can provide sufficient time for "slow down, evaluate, and avoid" to maximize the efficiency of the surgical procedure.

[0097] The various surgical visualization systems disclosed herein may not require a contrast agent or dye to be injected into tissue. For example, spectral imaging is configured to visualize hidden structures during surgery without using a contrast agent or dye. In other examples, the contrast agent may be easier to inject into the appropriate layer(s) of tissue than other visualization systems. The time between injection of the contrast agent and visualization of the critical structure can be, for example, less than 2 hours.

[0098] The various surgical visualization systems disclosed herein may be linked to clinical data and / or device data. For example, the data can provide a boundary as to how close an energy-supplied surgical device (or other device that may cause damage) is to tissue that the surgeon does not want to damage. Any data module that interfaces with the surgical visualization systems disclosed herein can be provided integrally or separately with a robot, for example, to enable use with a stand-alone surgical device in an open or laparoscopic procedure. The surgical visualization systems can, in various examples, be compatible with robotic surgical systems. For example, the visualization image / information can be displayed within the robotic console.

[0099] In various examples, a clinician may not know the location of important structures relative to a surgical instrument. For example, when an important structure is embedded in tissue, the clinician may not be able to identify the location of the important structure. In certain examples, the clinician may desire to maintain the surgical device outside of a range of positions surrounding the important structure and / or away from visible tissue covering the hidden important structure. When the location of a hidden important structure is unknown, the clinician risks approaching the important structure too closely, resulting in inadvertent trauma and / or incision of the important structure and / or applying excessive energy, heat, and / or tension near the important structure. Alternatively, the clinician may remain too far away from a suspected location of the important structure, attempting to avoid the important structure and risking affecting tissue in less desirable locations.

[0100] A surgical visualization system is provided that presents tracking of a surgical device relative to one or more important structures. For example, the surgical visualization system can track the proximity of the surgical device relative to the important structure. Such tracking can occur during surgery, in real-time and / or near real-time. In various examples, the tracking data can be provided to the clinician via a display screen (e.g., a monitor) of an imaging system.

[0101] In one aspect of the present disclosure, a surgical visualization system includes a surgical device having an emitter configured to emit a structured light pattern onto a visible surface, an imaging system having a camera configured to detect an embedded structure and the structured light pattern on the visible surface, and a control circuit in signal communication with the camera and the imaging system. The control circuit is configured to determine a distance from the surgical device to the embedded structure and provide a signal indicative of the distance to the imaging system. For example, the distance can be determined based on a three-dimensional view of an illuminated structure provided by images from a plurality of lenses (e.g., a left lens and a right lens) of the camera by calculating the distance from the camera to an important structure that fluoresces by fluoroscopy. The distance from the surgical device to the important structure can be triangulated, for example, based on known positions of the surgical device and the camera. Alternative means for determining the distance to the embedded important structure are further described herein. For example, an NIR time-of-flight distance sensor can be employed. Additionally, or alternatively, the surgical visualization system can determine a distance to visible tissue that overlays / covers the embedded important structure. For example, the surgical visualization system can identify a hidden important structure and expand the view of the hidden important structure by depicting an outline of the hidden important structure on a visible structure, such as a line on the surface of the visible tissue. The surgical visualization system can further determine a distance to an extended line on the visible tissue.

[0102] As provided by the various surgical visualization systems disclosed herein, by providing clinicians with up-to-date information regarding the proximity of surgical devices to hidden critical structures and / or visible structures, clinicians can make more informed decisions regarding the placement of surgical devices relative to hidden critical structures. For example, a clinician can view in real time / during surgery the distance between a surgical device and a critical structure, and in certain instances, the imaging system can provide alerts and / or warnings as the surgical device is moved within a defined proximity and / or zone of a critical structure. In certain instances, the alerts and / or warnings can be provided when the trajectory of the surgical device indicates a potential collision with a “no fly” zone in the vicinity of a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm or more of a critical structure). In such instances, without the need for clinician monitoring regarding the suspected location of a critical structure and the proximity of a surgical device thereto, the clinician can maintain momentum throughout the surgical procedure. As a result, certain surgical procedures can be performed more quickly with fewer pauses / interruptions and / or with improved accuracy and / or certainty. In one aspect, a surgical visualization system can be utilized to detect tissue variations, such as variations within an organ, in order to differentiate between healthy tissue and tumor / cancerous / diseased tissue. Such a surgical visualization system can maximize the removal of diseased tissue while minimizing the removal of healthy tissue.

[0103] Surgical hub system The various visualization systems described herein, i.e., imaging systems, can be incorporated into a surgical hub system as exemplified in connection with FIGS. 17-19 and further detailed below.

[0104] Referring to FIG. 17, a computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that may include a remote server 2113. In one example, as shown in FIG. 17, a surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112, which are configured to communicate with each other and / or with the hub 2106. In some aspects, a surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to 1.

[0105] FIG. 18 shows an example of a surgical system 2102 being used to perform surgery on a patient lying on an operating table 2114 within an operating room 2116. The robotic system 2110 is used as part of the surgical system 2102 in the surgery. The robotic system 2110 includes a surgeon's console 2118, a patient-side cart 2120 (a surgical robot), and a surgical robot hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side cart 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 2124, which can be manipulated by the patient-side cart 2120 to orient the imaging device 2124. The surgical robot hub 2122 can be used to process the images of the surgical site and then display them to the surgeon through the surgeon's console 2118.

[0106] Other types of robotic systems can be easily adapted to be used with the surgical system 2102. Various examples related to robotic system surgical instruments suitable for use with the present disclosure are described in various U.S. patent applications incorporated herein by reference in the present disclosure.

[0107] Various examples related to cloud-based analysis executed by the cloud 2104 and suitable for use with the present disclosure are described in various U.S. patent applications incorporated herein by reference in the present disclosure.

[0108] In various aspects, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0109] The optical components of the imaging device 2124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0110] The one or more illumination sources can be configured to irradiate electromagnetic energy in the visible spectrum as well as the invisible spectrum. The visible spectrum is, in some cases, also referred to as the optical spectrum or emission spectrum and is a portion of the electromagnetic spectrum that is visible to the human eye (i.e., can be detected by the human eye) and may be referred to as visible light or simply light. A typical human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0111] The invisible spectrum (i.e., the non-emitting spectrum) is a part of the electromagnetic spectrum that is located below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum and these become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum and these become invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0112] In various aspects, the imaging device 2124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (stomach cameras), endoscopes, laryngoscopes, nasopharyngo-ureteroscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0113] In one aspect, the imaging device employs multispectral monitoring to distinguish topography from underlying structures. A multispectral image captures image data within a specific wavelength range across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments having sensitivity to specific wavelengths beyond the visible light range, such as IR, and light from ultraviolet. Spectral imaging enables the extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in various U.S. patent applications incorporated herein by reference in the present disclosure. Multispectral monitoring can be a useful tool for repositioning the surgical field after a surgical task for performing one or more of the above-described tests on the treated tissue has been completed.

[0114] It is self-evident that strict sterilization of the operating room and surgical instruments is necessary in any surgical procedure. The strict hygiene and sterilization conditions required in the "surgical site", i.e., the operating room or treatment room, require the highest possible sterility of all medical device apparatuses and instruments. As part of the above sterilization process, there is a need to sterilize anything that comes into contact with the patient or enters the sterile field, including the imaging device 2124 and its accessories and components. It will be understood that the sterile field can be considered a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be considered the area immediately surrounding a surgically prepared patient. The sterile field can include properly attired and scrubbed team members, as well as all equipment and fixtures within that area. In various aspects, the visualization system 2108 includes one or more imaging sensors strategically positioned with respect to the sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in FIG. 18. In one aspect, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. The various components of the visualization system 2108 are described in various U.S. patent applications incorporated herein by reference in the present disclosure.

[0115] As shown in FIG. 18, the primary display 2119 is disposed in the sterile field so as to be visible to the operator of the operating table 2114. In addition, the visualization tower 21121 is disposed outside the sterile field. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 that face opposite to each other. The visualization system 2108 guided by the hub 2106 is configured to adjust the information flow to the operators inside and outside the sterile field using the displays 2107, 2109, and 2119. For example, the hub 2106 can cause the visualization system 2108 to display a snapshot of the surgical site recorded by the imaging device 2124 on the non-sterile display 2107 or 2109 while maintaining a live video of the surgical site on the primary display 2119. The snapshot on the non-sterile display 2107 or 2109 can enable, for example, a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0116] In one aspect, the hub 2106 is also configured to send diagnostic inputs or feedback entered by a non-sterile operator in the visualization tower 21121 to the primary display 2119 within the sterile field so that it can be seen by the sterile operator on the operating table. In one example, the input can be in the form of a modification to a snapshot displayed on the non-sterile display 2107 or 2109 that can be sent by the hub 2106 to the primary display 2119.

[0117] Referring to FIG. 18, the surgical instrument 2112 is used as part of a surgical system 2102 in a surgical procedure. The hub 2106 is also configured to adjust the information flow in accordance with the display of the surgical instrument 2112, as described in various U.S. patent applications incorporated by reference in the present disclosure. Diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 21121 can be sent by the hub 2106 to the surgical instrument display 2115 within the sterile field, where it can be viewed by the operator of the surgical instrument 2112. Exemplary surgical instruments suitable for use with the surgical system 2102 are described in various U.S. patent applications incorporated by reference herein in the present disclosure.

[0118] FIG. 19 shows a computer-implemented interactive surgical system 2200. The computer-implemented interactive surgical system 2200 is similar to the computer-implemented interactive surgical system 2100 in many respects. Each surgical system 2200 includes at least one surgical hub 2236 that communicates with a cloud 2204 that may include a remote server 2213. In one aspect, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to a plurality of surgical site devices such as, for example, intelligent surgical instruments, robots, and other computerized devices in the operating room. The surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. As shown in the embodiment of FIG. 19, the surgical hub 2236 is connected to an imaging module 2238 connected to an endoscope 2239, a generator module 2240 connected to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communication module 2230, a processor module 2232, a storage array 2234, a smart device / instrument 2235 optionally connected to a display 2237, and a non-contact sensor module 2242. The surgical site devices are connected to cloud computing resources and data storage via the surgical hub 2236. A robot hub 2222 may also be connected to the surgical hub 2236 and cloud computing resources. As described herein, in particular, the device / instrument 2235, the visualization system 2209 can be connected to the surgical hub 2236 via a wired or wireless communication standard or protocol. The surgical hub 2236 can be connected to a hub display 2215 (e.g., a monitor, a screen) to display and overlay images received from the imaging module, the device / instrument display, and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with the images and overlaid images.

[0119] Situation awareness Various visualization systems described herein, or aspects of visualization systems, can be utilized as part of a situation awareness system that can be embodied or executed by surgical hubs 2106, 2236 (Figs. 17 - 19). Specifically, when characterizing, identifying, and / or visualizing surgical instruments, or other surgical devices (including their positions, orientations, actions), tissue, structures, users, and other things located within the surgical field or operating room, context data can be provided that can be utilized by the situation awareness system to infer things such as the type of surgical procedure or step being performed, the type of tissue(s) and / or structure(s) being manipulated by the surgeon, etc. This context data can then be utilized by the situation awareness system to enable things such as providing alerts to the user, suggesting subsequent steps or actions for the user to take, predicting the use of surgical devices in preparation (e.g., predicting that an electrosurgical instrument will be used in a subsequent step of a surgical procedure and activating an electrosurgical generator), intelligent control of surgical instruments (e.g., customizing the operating parameters of a surgical instrument based on the specific health profile of each patient), etc.

[0120] "Intelligent" devices that include control algorithms that respond to sensed data can provide an improvement over "dumb" devices that operate without considering the sensed data. However, some sensed data may be incomplete or inconclusive when considered alone, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowledge of the surgical context (e.g., without knowledge of the type of tissue being operated on or the type of procedure being performed), the control algorithm may inaccurately or sub-optimally control the modular device with specific sensed data without context. Modular devices can include any surgical device that can be controlled by a situation awareness system, such as a visualization system device (e.g., a camera or a display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a smoke evacuator). For example, the optimal manner of a control algorithm for controlling a surgical instrument in response to a specific sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., resistance to tearing) and thus respond differently to actions taken by the surgical instrument. Therefore, even when the same measurement is sensed for a specific parameter, it may be desirable for the surgical instrument to take different actions. As a specific example, in response to the instrument sensing an unexpectedly high force to close its end effector, the optimal manner of controlling a surgical stapling and cutting instrument may vary depending on whether the tissue type is susceptible to tearing or resistant to it. In the case of tissue that is susceptible to tearing, such as lung tissue, the control algorithm of the instrument optimally decelerates the motor in response to the unexpectedly high force to close in order to avoid tearing the tissue. In the case of tissue that is resistant to tearing, such as stomach tissue, the control algorithm of the instrument optimally accelerates the motor in response to the unexpectedly high force to close in order to ensure that the end effector is properly clamped to the tissue.If it is not known whether the lung tissue or the stomach tissue is clamped, the control algorithm may make a suboptimal decision.

[0121] One solution utilizes a surgical hub that derives information about a surgical procedure being performed based on data received from various data sources and then appropriately controls paired modular devices. In other words, the surgical hub infers information about the surgical procedure from the received data and then is configured to control the modular devices paired with the surgical hub based on the inferred context about the surgical procedure. FIG. 20 shows a diagram of a situation-aware surgical system 2400 according to at least one aspect of the present disclosure. In some examples, the data source 2426 includes, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database including patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0122] The surgical hub 2404 may be similar to the hub 106 in many respects. For example, it can be configured to derive context information regarding a surgical procedure from a particular combination(s) of received data, or based on a particular order in which data is received from the data source 2426. The context information inferred from the received data can include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure being carried out by the surgeon, the type of tissue being operated on, or the body cavity being treated. This function related to some aspects of the surgical hub 2404 for deriving or inferring information regarding a surgical procedure from the received data is sometimes referred to as "situation awareness". In one example, the surgical hub 2404 can incorporate a situation awareness system that is hardware and / or programming associated with the surgical hub 2404 for deriving context information related to a surgical procedure from the received data.

[0123] The situation recognition system of the surgical hub 2404 can be configured to derive context information from data received from various different data sources 2426. In one example, the situation recognition system correlates various inputs (e.g., data from the database 2422, the patient monitoring device 2424, and / or the modular device 2402) with corresponding context information regarding the surgical procedure using a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained with training data. In other words, the machine learning system can be trained to accurately derive context information regarding the surgical procedure from the provided inputs. In another example, the situation recognition system can include a lookup table that stores pre-characterized context information regarding the surgical procedure in association with one or more inputs (or a range of inputs) corresponding to that context information. In response to a query by one or more inputs, the lookup table can return the corresponding context information of the situation recognition system to control the modular device 2402. In one example, the context information received by the situation recognition system of the surgical hub 2404 is associated with specific control adjustments, or a series of control adjustments, of one or more modular devices 2402. In another example, the situation recognition system includes a further machine learning system, lookup table, or other such system that generates or reads one or more control adjustments of one or more modular devices 2402 when context information is provided as an input.

[0124] The surgical hub 2404 incorporating the situation awareness system brings many advantages to the surgical system 2400. One advantage includes improving the interpretation of the detected and collected data, which improves the processing accuracy during the surgical procedure and / or the use of the data. Returning to a previous example, the situation awareness surgical hub 2404 can determine what type of tissue is being operated on, and thus, if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situation awareness surgical hub 2404 can correctly accelerate or decelerate the motor of the surgical instrument according to the tissue type.

[0125] As another example, the type of tissue being operated on can affect the adjustments made to the compression speed and load threshold of a surgical stapling and cutting instrument for specific tissue gap measurements. The situation awareness surgical hub 2404 can infer whether the surgical procedure being performed is a thoracic surgery or an abdominal surgery, whereby the surgical hub 2404 can determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is the lung (in the case of thoracic surgery) or the stomach (in the case of abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression speed and load threshold of the surgical stapling and cutting instrument according to the tissue type.

[0126] As yet another example, the type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situation awareness surgical hub 2404 can determine whether the surgical site is under pressure (by determining that the surgery is using insufflation), and determine the type of procedure. Generally, since a certain type of procedure is performed within a specific body cavity, the surgical hub 2404 can appropriately control the motor speed of the smoke evacuator according to the body cavity being operated on. Thus, the situation awareness surgical hub 2404 can provide a consistent amount of smoke evacuation for both thoracic and abdominal surgeries.

[0127] As yet another example of an implementation, the type of procedure being performed can affect the energy level that is optimal for the operation of an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument. For example, in arthroscopic procedures, the end effector of an ultrasonic surgical instrument or an RF electrosurgical instrument is immersed in fluid, and thus requires a higher energy level. The situational awareness surgical hub 2404 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 can then adjust the RF power level or ultrasonic amplitude (i.e., the “energy level”) of the generator to compensate for the fluid-filled environment. In connection therewith, the type of tissue being operated on can affect the energy level that is optimal for the operation of an ultrasonic surgical instrument or an RF electrosurgical instrument. The situational awareness surgical hub 2404 can determine which type of surgical procedure is being performed and then customize the energy levels of the ultrasonic surgical instrument or the RF electrosurgical instrument, respectively, according to the tissue profile expected for the surgical procedure. Further, the situational awareness surgical hub 2404 can be configured to adjust the energy levels of the ultrasonic surgical instrument or the RF electrosurgical instrument not only for each procedure but also over the course of the surgical procedure. The situational awareness surgical hub 2404 can determine which step of the surgical procedure is being performed or will be performed next and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or the RF electrosurgical instrument to set the energy level to a value appropriate for the tissue type expected according to the step of the surgical procedure.

[0128] As yet another example, in order for the surgical hub 2404 to improve conclusions drawn from one data source 2426, it is also possible to derive data from additional data sources 2426. The situation-aware surgical hub 2404 can enhance data received from the modular device 2402 with context information constructed regarding the surgical procedure from other data sources 2426. For example, the situation-aware surgical hub 2404 can be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, the video or image data may not be conclusive. Thus, in one example, the surgical hub 2404 further compares physiological measurements (e.g., blood pressure detected by a BP monitor communicatively coupled to the surgical hub 2404) with visual or image data of hemostasis (e.g., from the medical imaging device 124 (FIG. 2) communicatively coupled to the surgical hub 2404) to make a determination regarding the integrity of the staple line or tissue weld. In other words, the situation-aware system of the surgical hub 2404 can provide additional context when analyzing the visualization data, taking into account the physiological measurement data. The additional context can be useful when the visualization data may not be conclusive or may be incomplete by itself.

[0129] As another advantage, in order to reduce the number of times required for medical personnel to interact with or control the surgical system 2400 during the course of a surgical procedure, the paired modular devices 2402 are actively and automatically controlled according to specific steps of the surgical procedure being performed. For example, the situation-aware surgical hub 2404 can actively activate the generator to which the RF electrosurgical instrument is connected if it is determined that the use of the instrument is required in a subsequent step of the procedure. By actively activating the energy source, the instrument can be made ready for use as soon as the preceding step of the procedure is completed.

[0130] As another example, the situation awareness surgery hub 2404 can determine whether the current or subsequent steps of a surgical procedure require different views or magnifications on the display according to the shape part(s) of the surgical site that the surgeon is expected to view. The surgery hub 2404 can then appropriately and actively change the displayed view (e.g., supplied from a medical imaging device for the visualization system 108), such that the display automatically adjusts throughout the surgical procedure.

[0131] As yet another example, the situation awareness surgery hub 2404 can determine which step of a surgical procedure is being performed or will be performed next, and whether a particular data or comparison of data is required for the corresponding step of the surgical procedure. The surgery hub 2404 can be configured to automatically call up a data screen based on the step of the surgical procedure being performed, without waiting for the surgeon to request specific information.

[0132] Another advantage is that errors can be checked during the surgical setup or during the surgical procedure. For example, the situation awareness surgical hub 2404 can determine whether the surgical site is properly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 can determine the type of surgical procedure being performed, read out the corresponding checklist, product location, or setup requirements (e.g., from memory), and then be configured to compare the current layout of the surgical site with the standard layout for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 can be configured to compare, for example, a list of items for a procedure scanned by a suitable scanner and / or a list of devices paired with the surgical hub 2404 with the recommended or predicted inventory of items and / or devices for a given surgical procedure. If there are discrepancies between the lists, the surgical hub 2404 can be configured to provide an alert indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing. In one example, the surgical hub 2404 can be configured to determine, for example, the relative distance or relative position of the modular device 2402 and the patient monitoring device 2424 by means of a proximity sensor. The surgical hub 2404 can compare the relative position of the devices with the layout recommended or predicted for a particular surgical procedure. If there are discrepancies between the layouts, the surgical hub 2404 can be configured to provide an alert indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0133] As another example, the situation awareness surgery hub 2404 can determine whether a surgeon (or other healthcare provider) is making a mistake or deviating from a series of actions expected during a surgical procedure. For example, the surgery hub 2404 can determine the type of surgical procedure being performed, read a corresponding list of steps or order of device use (e.g., from memory), and then compare the steps being performed or devices being used during the surgical procedure to the steps or devices expected for the type of surgical procedure that the surgery hub 2404 has determined is being performed. In one example, the surgery hub 2404 can be configured to provide an alert indicating that an unexpected action is being performed or an unexpected device is being utilized during a particular step in the surgical procedure.

[0134] Overall, the situation awareness system for the surgery hub 2404 adjusts the surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure (e.g., adjusts for different tissue types) and verifies actions during the surgical procedure to improve the outcome of the surgical procedure. The situation awareness system also improves the efficiency of the surgeon during the surgical procedure by automatically suggesting the next steps, providing data, and adjusting the displays and other modular devices 2402 within the surgical site according to the specific context of the procedure.

[0135] Referring now to FIG. 21, for example, a timeline 2500 is shown that illustrates the situation awareness of a hub such as surgery hub 106, or 206 (FIGS. 1 - 11). The timeline 2500 shows an exemplary surgical procedure and the context information that the surgery hubs 106, 206 can derive from data received from data sources at each step of the surgical procedure. The timeline 2500 shows the general steps that a nurse, surgeon, and other healthcare providers would take during a lobectomy procedure that begins with the setup of the surgical site and ends with transferring the patient to the post - operative recovery room.

[0136] The situation awareness surgical hubs 106, 206 receive data from a data source that includes data generated each time a healthcare provider utilizes a modular device paired with the surgical hubs 106, 206 throughout the course of a surgical procedure. The surgical hubs 106, 206 receive this data from the paired modular devices and other data sources, and can continuously derive inferences regarding the ongoing procedure (i.e., context information), such as which step of the procedure is being performed at any given time, when new data is received. The situation awareness system of the surgical hubs 106, 206 can, for example, record data regarding the procedure to generate a report, verify the steps being performed by the healthcare provider, provide data or prompts that may be relevant to a particular surgical step (e.g., via a display screen), adjust the modular device based on the context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument, etc.), and perform any of these other actions described above.

[0137] As a first step 2502 in this exemplary procedure, a hospital staff member reads the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hubs 106, 206 determine that the procedure to be performed is a chest surgery.

[0138] In a second step 2504, the staff member scans the medical supplies that have arrived for the procedure. The surgical hubs 106, 206 cross-reference the scanned supplies with a list of supplies utilized in various types of procedures and confirm that the combination of supplies matches a chest procedure. Additionally, the surgical hubs 106, 206 can also determine that the surgery is not a wedge procedure (either because the supplies that have arrived do not include the specific supplies required for a chest wedge surgery or for some other reason that the supplies do not correspond to a chest wedge surgery).

[0139] In step 3 (2506), the healthcare provider scans the patient band via a scanner communicatively connected to the surgical hubs 106, 206. The surgical hubs 106, 206 can then confirm the patient's identification information based on the scanned data.

[0140] In step 4 (2508), the healthcare staff turns on the auxiliary devices. The auxiliary devices being utilized can vary according to the type of surgical procedure and the techniques used by the surgeon, but in this exemplary case, include a smoke evacuator, an air insufflator, and a medical imaging device. When the auxiliary devices are activated, the auxiliary devices, which are modular devices, can automatically pair with the surgical hubs 106, 206 located within a specific vicinity of the modular devices as part of their initialization process. The surgical hubs 106, 206 can then derive context information regarding the surgical procedure by detecting the type of modular device being paired during this pre-operative or initialization phase. In this specific example, the surgical hubs 106, 206 determine that the surgical procedure is a VATS procedure based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, a list of medical supplies to be used in the procedure, and the type of modular devices connected to the hubs, the surgical hubs 106, 206 can generally infer the specific procedure being performed by the surgical team. When the surgical hubs 106, 206 recognize which specific procedure is being performed, the surgical hubs 106, 206 then read the steps of that surgery from memory or from the cloud and then cross-reference the data continuously received from the connected data sources (e.g., modular devices, and patient monitoring devices) to infer which step of the surgical procedure the surgical team is performing.

[0141] In the fifth step 2510, the staff attaches the EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hubs 106, 206. When the surgical hubs 106, 206 start receiving data from the patient monitoring devices, the surgical hubs 106, 206 confirm that the patient is at the surgical site.

[0142] In the sixth step 2512, the medical staff anesthetizes the patient. The surgical hubs 106, 206 can infer that the patient is under anesthesia based on data from modular devices, and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or combinations thereof. When the sixth step 2512 is completed, the preoperative part of the pneumonectomy is completed and the surgical part begins.

[0143] In the seventh step 2514, the lung of the patient being operated on is deflated (while switching ventilation to the contralateral lung). The surgical hubs 106, 206 can infer, for example, from the ventilator data that the patient's lung has been deflated. Since the surgical hubs 106, 206 can compare the detection of the deflation of the patient's lung with the expected steps of the procedure (which can be accessed or read in advance), it can be inferred that the surgical part of the procedure has started, and thereby it can be determined that deflating the lung is the first surgical step in this particular procedure.

[0144] In step 8, 2516, a medical imaging device (e.g., a scope) is inserted and video shooting from the medical imaging device is started. The surgical hubs 106, 206 receive medical imaging device data (i.e., video data or image data) through the connection to the medical imaging device. When receiving the medical imaging device data, the surgical hubs 106, 206 can determine that the laparoscopic part of the surgery has started. Further, the surgical hubs 106, 206 can determine that the specific procedure being performed is a segmentectomy rather than a lobectomy (note that based on the data received in step 2, 2504 of the surgery, wedge surgery has already been determined by the surgical hubs 106, 206 to have no possibility). Using the data from the medical imaging device 124 (Figure 2), it is possible to determine the angle at which the medical imaging device is directed with respect to the visualization of the patient's anatomical structure, monitor the number of medical imaging devices being used (i.e., activated and paired with the surgical hubs 106, 206), and monitor the type of visualization device being used, so as to determine context information regarding the type of procedure being performed in several different ways. For example, one technique for performing VATS lobectomy is to place the camera above the diaphragm at the anteroinferior corner of the patient's chest cavity, while another technique for performing VATS segmentectomy is to place the camera at an intercostal position anterior to the segmental fissure. The situation recognition system can be trained, for example, using pattern recognition techniques or machine learning techniques, to recognize the position of the medical imaging device according to the visualization of the patient's anatomical structure. As another example, one technique for performing VATS lobectomy uses a single medical imaging device, while another technique for performing VATS segmentectomy uses multiple cameras. As yet another example, one technique for performing VATS segmentectomy uses an infrared light source (which can be communicably connected to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in VATS lobectomy.By tracking any or all of this data from the medical imaging device, the surgical hubs 106, 206 can determine the specific type of surgical procedure being performed and / or the techniques being used in a particular type of surgical procedure.

[0145] In the ninth step 2518, the surgical team begins the incision step of the procedure. Since the surgical hubs 106, 206 receive data from the RF or ultrasonic generator indicating that the energy instrument is being fired, it can be inferred that the surgeon is in the process of incising and moving the patient's lung. The surgical hubs 106, 206 can determine that the energy instrument being fired at this point in the process (i.e., after the steps of the procedure described above are completed) corresponds to the incision step by cross-referencing the received data with the read steps of the surgical procedure. In a particular example, the energy instrument can be an energy instrument attached to the robotic arm of a robotic surgical system.

[0146] In the tenth step 2520, the surgical team proceeds to the ligation step of the procedure. Since the surgical hubs 106, 206 receive data from the surgical stapling and cutting instrument indicating that the instrument is being fired, it can be inferred that the surgeon is currently ligating the arteries and veins. Similar to the previous step, the surgical hubs 106, 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the read steps of the process. In a particular example, the surgical instrument can be a surgical tool attached to the robotic arm of a robotic surgical system.

[0147] In the 11th step 2522, an excision of the treatment area is performed. The surgical hubs 106, 206 can be inferred by the surgeon to be cutting across the substance based on data from the surgical stapling and cutting instruments, including data from its cartridge. The data of the cartridge can correspond to, for example, the size or type of staples being fired by the instrument. Since different types of staples are used for different types of tissue, the data of the cartridge can indicate the type of tissue being stapled and / or cut across. In this case, the type of staple being fired is used for the substance (or other similar tissue types), whereby the surgical hubs 106, 206 can be inferred that an excision of the treatment area is being performed.

[0148] Subsequently, in the 12th step 2524, a nodule incision step is performed. The surgical hubs 106, 206 can be inferred by the surgical team to be incising the nodule and performing a leak test based on data received from a generator indicating that an RF or ultrasonic instrument is being fired. In this particular treatment, the RF or ultrasonic instrument used after the substance tissue has been cut across corresponds to the nodule incision step, whereby the surgical hubs 106, 206 are able to make the above inference. Since different instruments are better suited for specific tasks, it should be noted that the surgeon can periodically alternate between the surgical stapling / cutting instrument and the surgical energy (i.e., RF or ultrasonic) instrument according to the specific steps during the treatment. Therefore, the specific sequence in which the stapling / cutting instrument and the surgical energy instrument are used can indicate which step of the treatment the surgeon is performing. Further, in a specific example, robotic tools can be used for one or more steps in the surgery and / or handheld surgical instruments can be used for one or more steps in the surgery. The surgeon(s) can, for example, alternately use the robotic tool and the handheld surgical instrument and / or use the devices simultaneously. When the 12th step 2524 is completed, the incision is closed and the postoperative part of the treatment begins.

[0149] In step 2526 of operation 13, the patient is awakened from anesthesia. The surgical hubs 106, 206 can, for example, infer that the patient is starting to wake up from anesthesia based on ventilator data (i.e., the patient's respiratory rate starting to increase).

[0150] Finally, in step 2528 of operation 14, healthcare personnel remove various patient monitoring devices from the patient. Thus, the surgical hubs 2106, 2236 can infer that the patient is being transferred to the recovery room when the hubs lose EKG data, BP data, and other data from the patient monitoring devices. As can be seen from this illustrative surgery description, based on the data received from the various data sources communicatively coupled to the surgical hubs 2106, 2236, the surgical hubs 2106, 2236 can determine or infer when each step of a given surgical procedure is being performed.

[0151] Situational awareness is further described in various U.S. patent applications incorporated herein by reference, the entire disclosures of which are incorporated herein by reference. In certain examples, for instance, the operation of a robotic surgical system, including various robotic surgical systems disclosed herein, can be controlled by the hubs 2106, 2236 based on its situational awareness and / or feedback from its components and / or information from the cloud 2104 (FIG. 17).

[0152] FIG. 22 is a logical flow diagram of a process 4000 showing a control program or logical configuration for correlating visualization data and instrument data according to at least one aspect of the present disclosure. The process 4000 is generally executed during a surgical procedure and includes, at 4001, receiving or deriving a first data set, i.e., visualization data, showing a visual aspect of a surgical instrument with respect to the surgical field, and at 4002, receiving or deriving a second data set, i.e., instrument data, showing a functional aspect of the surgical instrument, and at 4003, correlating the first data set and the second data set.

[0153] In at least one example, correlating visualization data with instrument data is implemented by constructing a composite data set from the visualization data and the instrument data. Process 4000 may further include comparing the composite data set with another composite data set that can be received from an external source and / or derived from a previously collected composite data set. In at least one example, Process 4000 includes displaying a comparison of the two composite data sets, as described in more detail below.

[0154] The visualization data of Process 4000 can show the visual aspects of the end effector of a surgical instrument with respect to tissue within the surgical field. Additionally or alternatively, the visualization data can show the visual aspects of the tissue being treated by the end effector of the surgical instrument. In at least one example, the visualization data represents one or more positions of the end effector or one of its components with respect to tissue within the surgical field. Additionally or alternatively, the visualization data can represent one or more movements of the end effector or one of its components with respect to tissue within the surgical field. In at least one example, the visualization data represents one or more changes in the shape, dimensions, and / or color of the tissue being treated by the end effector of the surgical instrument.

[0155] In various aspects, the visualization data is derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). The visualization data can be derived from various measurements, readings, and / or any other suitable parameters monitored and / or captured by the surgical visualization system, as will be described in more detail in connection with FIGS. 1-18. In various examples, the visualization data represents one or more visual aspects of the tissue within the surgical field and / or one or more visual aspects of the surgical instrument with respect to the tissue within the surgical field. In a particular example, the visualization data represents or identifies the position and / or movement of the end effector of the surgical instrument with respect to a critical structure (e.g., critical structure 101 in FIG. 1) within the surgical field. In a particular example, the visualization data is derived from surface mapping data, imaging data, tissue identification data, and / or distance data calculated by surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logics 136, 138, 140, 141.

[0156] In at least one example, the visualization data is derived from tissue identification and geometric surface mapping performed by visualization system 100 in combination with distance sensor system 104, as will be described in more detail in connection with FIG. 1. In at least one example, the visualization data is derived from measurements, readings, or any other sensor data captured by imaging device 120. As described in connection with FIG. 1, imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of an image based on the molecular response to various wavelengths.

[0157] Additionally, or alternatively, the visualization data can be derived from measurements, readings, or any suitable sensor data captured by any suitable imaging device, such as a camera or imaging sensor configured to detect, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In at least one example, as described in more detail in connection with FIGS. 3-4 and FIGS. 13-16, the visualization data is derived from a visualization system 160 that includes a light waveform emitter 123 and a waveform sensor 122 configured to detect the reflected waveform. In yet another example, the visualization data is derived from a visualization system that includes a three-dimensional (3D) camera and associated electronic processing circuitry, such as, for example, visualization system 500. In yet another example, the visualization data is derived from a structured (or patterned) light system 700, which is described in more detail in connection with FIG. 12. The foregoing examples can be used alone or in combination to derive the visualization data for process 4000.

[0158] The instrument data for process 4000 can indicate the operation of one or more internal components of a surgical instrument. In at least one example, the instrument data represents one or more operating parameters of an internal component of the surgical instrument. The instrument data can represent the position and / or movement of one or more internal components of the surgical instrument. In at least one example, the internal component is a cutting member configured to cut tissue during a firing sequence of the surgical instrument. Additionally, or alternatively, the internal component can include one or more staples configured to be fired into tissue during a firing sequence of the surgical instrument.

[0159] In at least one example, the instrument data represents one or more operations of one or more components of one or more drive assemblies of a surgical instrument, such as, for example, an articulation drive assembly, a closure drive assembly, a rotation drive assembly, and / or a firing drive assembly. In at least one example, the instrument data set represents one or more operations of one or more drive members of a surgical instrument, such as, for example, an articulation drive member, a closure drive member, a rotation drive member, and / or a firing drive member.

[0160] FIG. 23 is a schematic view of an exemplary surgical instrument 4600 for use with a process 4000 that is in many respects similar to other surgical instruments or tools described by this disclosure, such as, for example, surgical instrument 2112. For the sake of brevity, various aspects of process 4000 are described only by this disclosure using a handheld surgical instrument. However, this is not limiting. Such aspects regarding process 4000 can be equally implemented with a robotic tool, such as, for example, surgical tool 2117.

[0161] Surgical instrument 4600 includes a plurality of motors that can be activated to perform various functions. The plurality of motors of surgical instrument 4600 can be activated to cause a firing motion, a closure motion, and / or an articulation motion at the end effector. The firing motion, the closure motion, and / or the articulation motion can be transmitted, for example, via a shaft assembly, to the end effector of surgical instrument 4600. However, in other examples, a surgical instrument for use with process 4000 can be configured to manually perform one or more of a firing motion, a closure motion, and an articulation motion. In at least one example, surgical instrument 4600 includes an end effector that treats tissue by deploying staples into the tissue. In another example, surgical instrument 4600 includes an end effector that treats tissue by applying therapeutic energy to the tissue.

[0162] In a particular example, the surgical instrument 4600 includes a firing motor 4602. The firing motor 4602 can be operably coupled to a firing motor drive assembly 4604 configured to transmit the firing motion generated by the firing motor 4602 to the end effector, specifically, for example, to move a firing member in the form of an I-beam that may include a cutting member. In a particular example, the firing motion generated from the firing motor 4602 can deploy staples into tissue captured by the end effector from a staple cartridge and, optionally, advance an I-beam cutting member to cut, for example, the captured tissue.

[0163] In a particular example, the surgical instrument or tool may include a closure motor 4603. The closure motor 4603 can be operably coupled to a closure motor drive assembly 4605 configured to transmit the closure motion generated by the closure motor 4603 to the end effector, specifically, to displace a closure tube to close an anvil and compress tissue between the anvil and a staple cartridge. By the closure motion, the end effector can transition, for example, from an open configuration to an approaching configuration to capture tissue.

[0164] In a particular example, the surgical instrument or tool may include, for example, one or more articulation motors 4606a, 4606b. The articulation motors 4606a, 4606b can be operably coupled to respective articulation motor drive assemblies 4608a, 4608b configured to transmit the articulation motion generated by the articulation motors 4606a, 4606b to the end effector. In a particular example, by the articulation motion, for example, the end effector can articulate relative to the shaft.

[0165] In certain examples, a surgical instrument or tool may include a control module 4610 that can be employed with a plurality of motors of the surgical instrument 4600. Each of the motors 4602, 4603, 4606a, 4606b may include a torque sensor for measuring the output torque on the shaft of the motor. The force on the end effector may be sensed in any conventional manner, such as by a force sensor outside of the jaw or by a torque sensor of the motor that actuates the jaw.

[0166] In various examples, as shown in FIG. 23, the control module 4610 may include a motor driver 4626 that may include one or more H-bridge FETs. The motor driver 4626 can modulate the power transmitted from a power source 4628 to a motor coupled to the control module 4610 based on an input from, for example, a microcontroller 4620 (the "controller"). In certain examples, as described above, the controller 4620 can be employed to determine, for example, the current drawn by the motor while the motor is coupled to the control module 4610.

[0167] In certain examples, the microcontroller 4620 may include a microprocessor 4622 (the "processor") and one or more non-transitory computer-readable media or memory units 4624 (the "memory"). In certain examples, the memory 4624 can store various program instructions that, when executed, can cause the processor 4622 to perform the plurality of functions and / or calculations described herein. In certain examples, one or more of the memory units 4624 can be coupled to, for example, the processor 4622. In various examples, the processor 4622 can control the motor driver 4626 to control the position, rotational direction, and / or speed of a motor coupled to the control module 4610.

[0168] In certain examples, for instance, one or more mechanisms and / or sensors, such as sensor 4630, can be configured to detect the force (closing force “FTC”) applied to tissue captured between the jaws by the jaws of the end effector of surgical instrument 4600. The FTC can be transmitted to the jaws of the end effector via the closing motor drive assembly 4605. Additionally, or alternatively, sensor 4630 can be configured to detect the force (firing force “FTF”) applied to the end effector through the firing motor drive assembly 4604. In various examples, sensor 4630 can be configured to detect closing operations (e.g., motor current and FTC), firing operations (e.g., motor current and FTF), articulation movement (e.g., angular position of the end effector), and rotation of the shaft and end effector.

[0169] One or more aspects of process 4000 can be executed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4000 are executed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4000. Additionally, or alternatively, one or more aspects of process 4000 can be executed by combinational logic circuits (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuits (e.g., control circuit 420 of FIG. 2C). Further, process 4000 can be executed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0170] In various aspects, process 4000 can be implemented by a computer-implemented interactive surgical system 2100 (FIG. 19) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that can include a remote server 2113. A control circuit that executes one or more aspects of process 4000 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can communicate with a surgical instrument (e.g., surgical instruments 2112, 4600) to receive instrument data therefrom. Communication between the surgical instrument and the control circuit of the visualization system can be direct communication, or instrument data can be sent to the visualization system, for example, through a surgical hub 2106. In at least one example, a control circuit that executes one or more aspects of process 4000 can be a component of a surgical hub 2106.

[0171] Referring to FIG. 24, in various examples, visualization data 4010 is correlated with instrument data 4011 by constructing a composite data set 4012 from visualization data 4010 and instrument data 4011. FIG. 24 shows, in graph 4013, a composite data set 4012 of a current user constructed from the current user's visualization data 4010 and the current user's instrument data 4011. Graph 4013 shows visualization data 4010 representing a first use cycle of surgical instrument 4600 involving joe positioning, clamping, and firing of surgical instrument 4600. Graph 4013 also shows visualization data 4010 representing the start of a second use cycle of surgical instrument 4600, in which case the joe has been repositioned for a second clamp and firing of surgical instrument 4600. Graph 4013 further shows the current user's instrument data 4011 in the form of FTC data 4014 correlated with the clamp visualization data and FTF data 4015 correlated with the firing visualization data.

[0172] As described above, the visualization data 4010 is derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and can, for example, represent the positioning, clamping, and / or firing of the end effector of the surgical instrument 4600 and the distance between the end effector of the surgical instrument 4600 and important structures within the surgical field. In at least one example, the visualization system identifies the end effector or its components within the surgical field, identifies important structures within the surgical field, and tracks the position of the end effector or its components relative to the important structures or the tissue surrounding the important structures. In at least one example, the visualization system identifies the jaws of the end effector within the surgical field, identifies important structures within the surgical field, and further tracks the position of the jaws relative to the important structures or the tissue surrounding the important structures within the surgical field.

[0173] In at least one example, the important structure is a tumor. To remove this tumor, a surgeon generally wants to cut the tissue along a safe margin around the tumor to ensure complete removal of the entire tumor. In such an example, the visualization data 4010 can represent the distance between the jaws of the end effector and the safe margin of the tumor during the positioning, clamping, and / or firing of the surgical instrument 4600.

[0174] Process 4000 may further include comparing a composite dataset 4012 of a current user with another composite dataset 4012' that can be received from an external source and / or derived from previously collected composite datasets. Graph 4013 shows a comparison between the composite dataset 4012 of the current user and another composite dataset 4012' that includes visualization data 4010', instrument data 4011' having FTC data 4014' and FTF data 4015'. This comparison can be presented to the user of the surgical instrument 4600 in real time in the form of graph 4013 or any other suitable format. The control circuit that executes one or more aspects of process 4000 can compare the two composite datasets on any suitable screen in the operating room, such as, for example, the screen of a visualization system. In at least one example, the comparison can be displayed alongside a real-time video of the surgical field captured on any suitable screen in the operating room. In at least one example, the control circuit is configured to adjust instrument parameters to address deviations detected between the first composite dataset and the second composite dataset.

[0175] Further, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4000 may cause the current state of instrument data, such as, for example, FTF data and / or FTC data, to be displayed relative to the best-in-class equivalent. In the example shown in FIG. 24, the current value of FTC represented by circle 4020 is shown in real time relative to gauge 4021, and indicator 4022 represents the best-in-class FTC. Similarly, the current value of FTF represented by circle 4023 is shown relative to gauge 4024, and indicator 4025 represents the best-in-class FTF. Such information can be overlaid in real time on the video image of the surgical field.

[0176] The example shown in FIG. 24 warns the user that the current FTC is higher than the best - practice FTC and that the current FTF is higher than the best - practice FTF. A control circuit that executes one or more aspects of process 4000 can warn the current user of surgical instrument 4600 with an audible, visual, and / or tactile warning mechanism when the current value of the FTF and / or FTC reaches and / or moves beyond a predetermined threshold value.

[0177] In certain examples, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4000 can further provide the current user of surgical instrument 4600 with predicted instrument data based on the current instrument data. For example, as shown in FIG. 24, the predicted FTF 4015” is determined based on the current value of the FTF and is further displayed on graph 4013 against the current FTF 4015 and previously collected FTFs. Additionally, or alternatively, as shown in FIG. 24, a projected FTF circle 4026 can be displayed against gauge 4024.

[0178] In various aspects, the previously collected composite data set and / or the best - practice FTF and / or FTC are determined from the use of previous surgical instruments 4600 in the same surgical procedure and / or other surgical procedures performed by the user, other users within the hospital, and / or users of other hospitals. Such data can be made available to a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4000, for example, by importing from cloud 104.

[0179] In various aspects, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4000 can visually overlay feedback measurements of tissue thickness, compression, and stiffness on a screen that displays a live video of surgical instrument 4600 in the surgical field when the jaws of the end effector begin to deform the tissue captured between the jaws during the clamp phase. The visual overlay correlates visualization data representing tissue deformation with changes in the clamping force over time. This correlation helps in confirming the user's selection of an appropriate cartridge, determining the firing start time, and determining a suitable firing speed. Further, an adaptive clamp algorithm can also be learned from the correlation. The change in the adaptive firing speed can be known by the measured force and the change in the movement of the tissue (e.g., principal strain, tissue slippage, etc.) adjacent to the jaws of surgical instrument 4600 while a gauge or meter communicating the result is overlaid on a screen that displays a live video of the end effector in the surgical field.

[0180] In addition to the above, the kinematics of surgical instrument 4600 can also be used to instruct the operation of the instrument for another use or to a user. The kinematics can be verified with an accelerometer, torque sensor, force sensor, motor encoder, or any other suitable sensor, and various force, velocity, and / or acceleration data of the surgical instrument or its components can be obtained and correlated with corresponding visualization data.

[0181] In various aspects, when it is detected from the visualization data and / or instrument data that a deviation from the best practice surgical technique has occurred, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4000 can present an alternative surgical technique. In at least one example, an adaptive display of the movement, force, tissue impedance of the instrument, and the results with the proposed alternative technique is shown. When the visualization data shows the detection of blood vessels and clip appliers within the surgical field, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4000 can further ensure the perpendicularity of the blood vessels to the clip appliers. The control circuit can suggest changing the position, orientation, and / or roll angle to achieve the desired perpendicularity.

[0182] In various aspects, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can execute process 4000 by comparing real-time visualization data with a preoperative planning simulation. The user can utilize a preoperative patient scan to simulate a surgical approach. The preoperative planning simulation can enable the user to follow a specific preoperative plan based on training runs. The control circuit can be configured to correlate the reference landmarks of the preoperative scan / simulation with the current visualization data. In at least one example, the control circuit can employ boundary tracking of the object to establish the correlation.

[0183] When a surgical instrument interacts with tissue and deforms the surface geometry, the change in surface geometry can be calculated as a function of the position of the surgical instrument. For a given change in the position of the surgical instrument upon contact with tissue, the corresponding change in the geometry of the tissue's geometry can depend on the subsurface structure within the tissue region with which the surgical instrument is in contact. For example, in a thoracic surgery, the change in the geometry of the tissue in a region that includes a partial structure of the airway is different from a region that has a partial structure of parenchymal tissue. Generally, the harder the substructure, the smaller the change in the geometry of the surface tissue in response to a given change in the position of the surgical instrument. In various aspects, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to calculate a running average of the change in the position of the surgical instrument with respect to a change in the surface geometry of a given patient to obtain patient-specific differences. Additionally, or alternatively, the calculated running average can be compared to a second set of previously collected data. In certain examples, a surface reference can be selected if the change in surface geometry is not measured for each change in instrument position. In at least one example, the control circuit can be configured to determine the position of the substructure based on the detected change in surface geometry in response to a given contact between the tissue region and the surgical instrument.

[0184] Further, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can be configured to maintain contact between a set instrument and tissue based on a correlation between the contact of the set instrument with the tissue and a change in the surface geometry of one or more tissues associated with the contact of the set instrument with the tissue. For example, the end effector of the surgical instrument 4600 can clamp the tissue between the jaws with a desired compression that sets the contact between the instrument and the tissue. The corresponding change in the tissue and surface geometry can be detected by a visualization system. Further, the control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can derive visualization data indicative of the change in the tissue and surface geometry associated with the desired compression. Further, the control circuit can automatically adjust the motor settings of the closure motor 4603 (FIG. 22) to maintain the change in the tissue and surface geometry associated with the desired compression. In this arrangement, a continuous interaction between the surgical instrument 4600 and the visualization system is required to maintain the change in the tissue and surface geometry associated with the desired compression by continuously adjusting the compression of the jaws on the tissue based on the visualization data.

[0185] In yet another example, if the surgical instrument 4600 is a robotic tool attached to a robotic arm of a robotic surgical system (e.g., robotic system 110), the robotic surgical system can be configured to automatically adjust one or more components of the robotic surgical system to maintain a set surface contact with the tissue based on visualization data derived from a detected change in the detected surface geometry of the tissue in response to the set surface contact with the tissue.

[0186] In various examples, the visualization data is used in combination with the measured instrument data to maintain contact between tissues either in terms of position or load control such that while the instrument is being moved relative to the tissue, the user can manipulate the tissue to apply a predefined load to the tissue. The user can specify that they want to maintain contact or pressure and use visual tracking of the instrument along with the internal load of the instrument to enable repositioning without changing fixed parameters.

[0187] Referring to FIGS. 25A and 25B, the screen 4601 of a visualization system (e.g., visualization systems 100, 160, 500, 2108) displays a real-time video image of the surgical field during a surgical procedure. The end effector 4642 of the surgical instrument 4600 includes, for example, jaws for clamping tissue near a tumor identified in the surgical field via, for example, an overlaid MRI image. The jaws of the end effector 4642 comprise an anvil 4643 and a channel for accommodating a staple cartridge. At least one of the anvil 4643 and the channel is movable relative to the other for capturing tissue between the anvil 4643 and the staple cartridge. The captured tissue is then stapled via staples 4644 deployable from the staple cartridge during a firing sequence of the surgical instrument 4600. Additionally, the captured tissue is cut via a cutting member 4645 advanced distally during the firing sequence but slightly after staple deployment.

[0188] As is apparent in FIG. 25A, during the firing sequence, the position and / or movement of the captured tissue and certain internal components of the end effector 4642, such as the staples 4644 and the cutting member 4645, may not be visible in the standard view 4640 of the live video on the screen 4601. The particular end effector includes windows 4641, 4653 that make the cutting member 4645 partially visible at the beginning and end of the firing sequence but not during the firing sequence. Thus, the user of the surgical instrument 4600 cannot track the progress of the firing sequence on the screen 4601.

[0189] FIG. 26 is a logical flow diagram of a process 4030 that shows a control program or logical configuration for synchronizing the movement of a virtual representation of an end effector component with the actual movement of the end effector component according to at least one aspect of the present disclosure. Process 4030 is generally executed during a surgical procedure and includes, at 4031, detecting the movement of internal components of the end effector during a firing sequence, and at 4032, presenting a virtual representation of this internal component on the end effector, for example, by overlaying it, and at 4033, synchronizing the movement of the virtual representation on screen 4601 with the detected movement of the internal component.

[0190] One or more aspects of process 4030 can be executed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by the present disclosure. In at least one example, one or more aspects of process 4030 are executed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4030. Additionally, or alternatively, one or more aspects of process 4030 can be executed by combinational logic circuits (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuits (e.g., control circuit 420 of FIG. 2C). Further, process 4030 can be executed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by the present disclosure.

[0191] In various examples, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4030 can receive instrument data indicative of the movement of internal components of end effector 4642 during a firing sequence. The movement of the internal components can be tracked, for example, using a conventional rotary encoder of firing motor 4602. In other examples, the movement of the internal components can be tracked by a tracking system employing an absolute positioning system. A detailed description of the absolute positioning system is described in U.S. Patent Application Publication No. 2017 / 0296213, published Oct. 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is hereby incorporated by reference in its entirety. In some examples, the movement of the internal components can be tracked using one or more position sensors that can comprise any number of magnetic sensing elements, such as magnetic sensors that are classified depending on, for example, measuring a total magnetic field or measuring vector components of a magnetic field.

[0192] In various aspects, process 4030 includes an overlay trigger. In at least one example, the overlay trigger can detect that tissue has been captured by end effector 4642. When tissue captured by end effector 4642 is detected, process 4030 superimposes a virtual representation of cutting member 4645 at a starting position onto end effector 4642. Process 4030 further includes projecting a staple line that outlines where staples will be deployed within the captured tissue. Further, in response to activation of the firing sequence by a user, process 4030 moves the virtual representation of cutting member 4645 distally, mimicking the actual movement of cutting member 4645 within end effector 4642. When the staples are deployed, process 4030 converts unfired staples to fired staples so that the user can visually track staple deployment and the advancement of cutting member 4645 in real time.

[0193] In various examples, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4030 can detect that tissue has been captured by end effector 4642 from instrument data indicating, for example, the force that closing motor 4603 (FIG. 22) applies to the jaws of end effector 4642 through closing motor drive assembly 4605. The control circuit can further determine the position of the end effector within the operative field from visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108). In at least one example, the position of the end effector can be determined relative to a reference point within the tissue, such as a critical structure.

[0194] In any case, the control circuit superimposes a virtual representation of the internal component at a position corresponding to the position of the internal component within the end effector over the end effector 4642 on the screen 4601. Further, the control circuit moves the projected virtual representation of the internal component in synchronization with the internal component during the firing sequence. In at least one example, the synchronization is improved by the integration of markings on the end effector 4642 that can be used by the control circuit as a reference point for determining where to overlay the virtual representation.

[0195] Figure 25B shows an enlarged view 4651 of the live image of the surgical field on the screen 4601. In the enlarged view 4651 of the example of Figure 25B, the virtual displays of the staple 4644 and the cutting member 4645 are superimposed on the end effector 4642 during the firing sequence. The overlay differentiates between the fired staple 4644a and the unfired staple 4644b, and between the completed cutting line 4646a and the projected cutting line 4646b to track the progress of the firing sequence. Further, the overlay also shows the starting point of the staple line 4647 and the projected end 4649 of the staple line at the cutting line that does not reach the end of the tissue. And based on the overlay of the tumor MRI image, the safety margin distance "d" between the tumor and the projected cutting line 4646b is measured and presented together with the overlay. The superimposed safety margin distance "d" assures the user that all tumors are removed.

[0196] As shown in Figure 25B, the control circuit is configured to continuously relocate the virtual display of the internal components to the visualization system to correlate with the actual movement of the internal components. In the example of Figure 25B, this overlay shows that the completed cutting line 4646 is slightly lagging behind the fired staple line 4644a by a distance "d1", which assures the user that the firing sequence is progressing properly.

[0197] Referring now to Figure 27, a visualization system (e.g., visualization systems 100, 160, 500, 2108) can employ the instrument illumination 4058 and the camera 4059 to detect and / or define the trocar position. Based on the determined trocar position, the user can be directed to the most suitable trocar port to complete the intended function based on time efficiency, the position of critical structures, and / or avoidance or risk.

[0198] Figure 27 shows three trocar positions (Trocar 1, Trocar 2, Trocar 3) that extend through the body wall 4050 at different positions and orientations with respect to the body wall and to important structures 4051 within the cavity 4052 inside the body wall 4050. The trocars are represented by arrows 4054, 4055, 4056. The trocar positions can be detected by using a cascading light or image on the surrounding environment by the lighting device 4058. Further, the light source of the lighting device 4058 may be a rotary light source. In at least one example, the light sources of the lighting device 4058 and the camera 4059 are utilized, for example, to detect the distance of the trocar with respect to a target position such as the important structure 4051. In various examples, when a more preferable instrument position is determined based on visualization data derived from a camera 4059 that records the light projected by the light source of the lighting device 4058, the visualization system can suggest a change in the position of the instrument. The screen 4060 can display the distance between the trocar and the target tissue, whether instrument access through the trocar is acceptable, the risks associated with the use of the trocar, and / or the expected surgical time using the trocar, which can assist the user in selecting the optimal trocar for introducing the surgical instrument into the cavity 4052.

[0199] In various aspects, a surgical hub (e.g., surgical hubs 2106, 2122) can recommend an optimal trocar for inserting a surgical instrument into the cavity 4052 based on, for example, user characteristics that can be received from a user database. User characteristics include the user's dominant hand, the preferences of the user on the patient side, and / or the user's physical characteristics (e.g., height, arm length, range of motion). The surgical hub can utilize these characteristics, data on the positions and orientations of the available trocars, and / or position data of the important structures to select the optimal trocar for inserting the surgical instrument in order to reduce user fatigue and increase efficiency. In various aspects, the surgical hub can further reverse the control of the surgical instrument if the user reverses the orientation of the end effector.

[0200] The surgical hub can reconstruct the output of a surgical instrument based on visualization data. For example, if the visualization data indicates that the surgical instrument is being retracted or is being utilized to perform a different task, the surgical hub can suppress the activation of the treatment energy output of the surgical instrument.

[0201] In various aspects, the visualization system can be configured to track the blood surface or estimate the blood volume-based reflected IR or red wavelengths to depict the blood from non-blood surfaces and surface geometry measurements. This can be reported as an absolute static indicator or as a rate of change to provide quantitative data regarding the amount and degree of change of bleeding.

[0202] Referring to FIG. 28, for example, various elements of a visualization system (e.g., visualization systems 100, 160, 500, 2108) such as a structured light projector 706 and a camera 720 can be used to generate visualization data of an anatomical organ to generate a virtual 3D structure 4130 of the anatomical organ.

[0203] As described herein, for example, structured light in the form of stripes or lines is projected from a light source and / or projector 706 onto the surface 705 of a targeted anatomical structure to identify the shape and contour of the surface 705. For example, a camera 720, which can be similar to the imaging device 120 (FIG. 1) in various respects, can be configured to detect the projected light pattern on the surface 705. When the projected pattern is deformed upon hitting the surface 705, it becomes possible for the vision system to calculate the depth and surface information of the targeted anatomical structure.

[0204] FIG. 29 is a logical flow diagram of a process 4100 that shows a control program or logical configuration according to at least one aspect of the present disclosure. In various examples, the process 4100 identifies a surgical procedure at 4101 and identifies an anatomical organ targeted by the surgical procedure at 4102. Further, the process 4100 generates a virtual 3D structure 4130 of at least a portion of the anatomical organ at 4104, identifies an anatomical structure of at least a portion of the anatomical organ related to the surgical procedure at 4105, connects the anatomical structure to the virtual 3D structure 4130 at 4106, and superimposes a layout plan of the surgical procedure determined based on the anatomical structure onto the virtual 3D structure 4130 at 4107.

[0205] One or more aspects of the process 4100 can be executed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by the present disclosure. In at least one example, one or more aspects of the process 4100 are executed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of the process 4100. Additionally, or alternatively, one or more aspects of the process 4100 can be executed by combinational logic circuits (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuits (e.g., control circuit 420 of FIG. 2C). Further, one or more aspects of the process 4100 can be executed by any suitable circuit having any suitable hardware and / or software components that can be disposed in or associated with various suitable systems described by the present disclosure.

[0206] In various aspects, process 4100 can be implemented by a computer-implemented interactive surgical system 2100 (FIG. 19) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that can include a remote server 2113. A control circuit that executes one or more aspects of process 4100 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0207] A control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4100 can identify a surgical procedure at 4101 and / or identify an anatomical organ targeted by the surgical procedure at 4102 by obtaining such information from a database storing the information or directly from user input. In at least one example, the database is stored in a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). In at least one example, the database includes an in-hospital EMR.

[0208] In one aspect, the surgical system 2200 includes a surgical hub 2236 connected to a plurality of surgical site devices such as, for example, a visualization system (e.g., visualization systems 100, 160, 500, 2108) disposed in the operating room. In at least one example, the surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to a visualization system, the cloud 2204, and / or the remote server 2213. The control circuit of the surgical hub 2236 that executes one or more aspects of process 4100 can identify the surgical procedure in 4101 and / or identify the anatomical organ targeted by the surgical procedure in 4102 by obtaining such information from a database stored in the cloud 2204 or from the remote server 2213.

[0209] The control circuit that executes one or more aspects of process 4100 can cause the visualization system (e.g., visualization systems 100, 160, 500, 2108) to perform an initial scan of at least a portion of the anatomical organ to generate a three-dimensional (“3D”) structure 4130 of at least a portion of the anatomical organ targeted by the surgical procedure in 4104. In the example shown in FIG. 28, the anatomical organ is the stomach 4110. The control circuit can cause visualization data to be generated by utilizing structured light 4111 on one or more elements of the visualization system such as, for example, the structured light projector 706 and the camera 720, to perform a scan of at least a portion of the anatomical organ when the camera(s) is introduced into the body. The 3D structure of at least a portion of the anatomical organ can be generated by leveraging current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input.

[0210] Further, a control circuit that executes one or more aspects of process 4100 identifies, at 4105, the anatomical structure of at least a portion of an anatomical organ related to a surgical procedure. In at least one example, a user can select the anatomical structure using any suitable input device. Additionally or alternatively, the visualization system can include one or more imaging devices 120 with a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect a reflected spectral waveform and generate an image based on a molecular response to various wavelengths. The optical absorption characteristics or refractive characteristics of the tissue can be utilized by the control circuit to distinguish different tissue types of the anatomical organ, thereby identifying the associated anatomical structure. Additionally, the control circuit can utilize current visualization data, preoperative data (e.g., patient scans and other relevant clinical data), stored visualization data from previous similar surgical procedures performed on the same or other patients, and / or user input to identify the associated anatomical organ.

[0211] The identified anatomical structure can be the anatomical structure of the surgical field and / or the anatomical structure can be selectable by the user. In various examples, the position tracking of the associated anatomical structure can be extended beyond the current view of a camera directed at the surgical field. In one example, this is achieved by using a common visible connection landmark or by using secondary connection motion tracking. Secondary tracking can be achieved, for example, through a secondary imaging source, calculation of the movement of the scope, and / or a pre-established beacon that is a measurement by a secondary visualization system.

[0212] As detailed above in connection with FIG. 14, the visualization system can utilize a structured light projector 706 to cast an array of patterns or lines that a camera 720 can use to determine the distance to a target location. The visualization system can then emit a pattern or line of a known size at a set distance equal to the determined distance. Additionally, the spectral camera can determine the size of a pattern that can vary according to the light absorption, or refractive properties, of the tissue at the target location. The difference between the known size and the determined size indicates the tissue density at the target location, which indicates the tissue type at the target location. A control circuit that executes one or more aspects of process 4100 can identify the associated anatomical structure based at least in part on the determined tissue density at the target location.

[0213] In at least one example, a detected abnormality in tissue density can be associated with a diseased state. Further, the control circuit can select, update, or modify one or more settings of a surgical instrument that treats tissue based on the tissue density detected via the visualization data. For example, the control circuit can change various clamp parameters and / or firing parameters of a surgical stapler utilized to staple and cut tissue. In at least one example, the control circuit can be capable of slowing the firing sequence and / or extending the clamp time based on the tissue density detected by the visualization data. In various examples, the control circuit can warn the user of the surgical instrument of abnormal tissue density by displaying on the screen instructions, such as instructions to reduce bite size, instructions to increase or decrease the energy delivery output of an electrosurgical instrument, and instructions to adjust the amount of jaw closure. In another example, where the visualization data indicates that the tissue is adipose tissue, the instruction can be to increase power to shorten the energy application time.

[0214] Furthermore, identifying the type of surgical procedure can facilitate the identification of the target organ by the control circuit. For example, if the procedure is a left upper lobectomy, the lung is likely to be the target organ. Therefore, the control circuit only considers the visualization data and / or non-visualization data related to the lung and / or instruments commonly used in such procedures. And with the knowledge of the type of procedure, other image fusion algorithms that notify, for example, the location of a tumor or the placement of a staple line, become more effective.

[0215] In various aspects, knowledge of the position of the operating table and / or the pneumoperitoneum pressure is used by a control circuit that executes one or more aspects of process 4100 to establish a baseline position of the target anatomical organ and / or related anatomical structures identified from the visualization data. Movement of the operating table (e.g., moving the patient from a flat position to a reverse Trendelenburg position) can cause deformation of the anatomical structures, tracking this and comparing it to the baseline can continuously notify the position and state of the target organ and / or related anatomical structures. Similarly, changes in the pneumoperitoneum pressure within the body cavity can potentially interfere with the baseline visualization data of the target organ and / or related anatomical structures within the body cavity.

[0216] A control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can execute one or more aspects of a process that derives baseline visualization data of the target organ and / or related anatomical structures of a patient on an operating table during a surgical procedure, determines a change in the position of the operating table, and re-derives the baseline visualization data of the target organ and / or related anatomical structures of the patient at the new position.

[0217] Similarly, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can perform one or more aspects of a process that derives baseline visualization data of a target organ and / or related anatomical structures of a patient on an operating table during a surgical procedure, determines a change in the pneumoperitoneum pressure within the patient's body cavity, and re-derives the baseline visualization data of the patient's target organ and / or related anatomical structures at the new pneumoperitoneum pressure.

[0218] In various examples, as shown in FIG. 28, a control circuit that performs one or more aspects of process 4100 can couple a specified anatomical structure to a virtual 3D structure by overlaying landmarks or labels on the virtual 3D structure of the organ to indicate the position of the anatomical structure. The control circuit can also overlay user-defined structures and tissue planes on the virtual 3D structure. In various aspects, a hierarchy of tissue types can be established to organize the anatomical structures identified on the virtual 3D structure. Table 1 provided below shows an exemplary hierarchy of the lungs and the stomach.

[0219] [Table 1]

[0220] In various aspects, related anatomical structures identified on the virtual 3D structure can be renamed and / or repositioned by the user, either to correct an error or as desired. In at least one example, the correction can be voice-activated. In at least one example, the corrections are recorded for future machine learning.

[0221] In addition to the above, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4100 can, at 4107, overlay a surgical layout plan (e.g., layout plan 4120) onto a virtual 3D structure of a target organ (e.g., stomach 4110). In at least one example, the virtual 3D structure is displayed on a separate screen of the visualization system from a screen that displays a live video / view of the surgical field. In another example, one screen can alternately display the surgical field and a live video of the 3D structure. In such an example, the user can alternately switch between the two views using any suitable input device.

[0222] In the example shown in FIG. 28, the control circuit has determined that the surgery is a sleeve gastrectomy and the target organ is the stomach. In an initial scan of the abdominal cavity, the control circuit identifies the stomach, liver, spleen, greater curvature of the stomach, and pylorus using visualization data such as, for example, structured light data and / or spectral data. This is informed by knowledge of the procedure and the target structure.

[0223] For example, visualization data such as structured light data and / or spectral data can be utilized by the control circuit to identify the stomach 4110 by comparing the current structured light data with stored structured light data previously associated with such an organ. In at least one embodiment, the control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) can utilize structured light data representing characteristic anatomical contours of an organ and / or spectral data representing characteristics of characteristic subsurface tissue to identify anatomical structures relevant to a surgical layout plan (e.g., layout plan 4120).

[0224] In at least one example, visualization data can be used to identify the pyloric vein 4112 that indicates the position 4113 of the pylorus 4131, identify the gastro-colic vessel 4114 that indicates the position 4115 of the greater curvature 4110 of the stomach, identify the curvature 4116 of the right gastric vein that indicates the position 4117 of the incisura angularis 4132, and / or identify the position 4119 of the His angle 4121. The control circuit can assign landmarks to one or more of the identified positions. In at least one example, as shown in FIG. 28, the control circuit causes the visualization system to superimpose landmarks on positions 4113, 4117, 4119 on the virtual 3D structure of the stomach 4110 generated using the visualization data as described above. In various aspects, the landmarks can be superimposed synchronously on the virtual 3D structure and the operative field view to enable the user to switch views without losing sight of the landmarks. The user can zoom in on a view of the screen that displays the virtual 3D structure to show the overall layout plan, or can zoom in to show a portion similar to the operative field view. The control circuit can continuously track and update the landmarks.

[0225] In a sleeve gastrectomy, the surgeon staples the stomach tissue mainly about 4 cm from the pylorus. Prior to stapling, an energy device is introduced into the patient's abdominal cavity at the start of the sleeve gastrectomy procedure, and the gastroepiploic artery and peritoneum are incised from the greater curvature at a position about 4 cm or about 4 cm from the pylorus. As described above, the control circuit that identified the position 4113 can automatically cause an overlay of the end effector of the energy device at a position about 4 cm or about 4 cm from the position 4113. The start position of the sleeve gastrectomy can be identified by overlaying the end effector of the energy device at a position about 4 cm or about 4 cm from the pylorus, or any suitable landmark.

[0226] As the surgeon makes an incision along the greater curvature of the stomach, the control circuit causes the landmark at position 4113 and / or the overlaid end effector of the energy device to be removed. When the surgeon approaches the spleen, a distance indicator is automatically overlaid on the virtual 3D structure view and / or the operative field view. The control circuit can cause the distance indicator to specify a distance of 2 cm from the spleen. The control circuit can cause the distance indicator to blink and / or change color when the incision path reaches, for example, 2 cm from the spleen or is about to reach it. The distance indicator overlay remains until the user reaches position 4119 of His angle 4121.

[0227] Referring to FIG. 30, when a surgical stapler is introduced into the abdominal cavity, the control circuit can utilize the visualization data to identify the pylorus 4131, angular incisure 4132, greater curvature 4133 of the stomach 4110, lesser curvature 4134 of the stomach 4110, and / or other anatomical structures associated with sleeve gastrectomy. A busy overlay can be similarly shown. The introduction of a surgical instrument into the body cavity (e.g., introduction of a surgical stapler into the abdominal cavity) can be detected by the control circuit from visualization data indicating visual cues on the end effector, such as a unique color, label, and / or shape. The control circuit can identify the surgical instrument within a database that stores such visual cues and corresponding visual cues. Alternatively, the control circuit can prompt the user to identify the surgical instrument inserted into the body cavity. Alternatively, a surgical trocar that assists with access to the body cavity can include one or more sensors for detecting the surgical instrument inserted therethrough. In at least one example, the sensor comprises an RFID reader configured to identify the surgical instrument from an RFID chip of the surgical instrument.

[0228] In addition to landmarks that identify relevant anatomical structures, the visualization system can also overlay a surgical procedure layout plan 4135, which can be in the form of a recommended treatment pathway, onto the 3D critical structures and / or the operative field view. In the example of FIG. 30, the surgical procedure is a sleeve gastrectomy, and the treatment layout plan 4135 is in the form of three resection paths 4136, 4137, 4138, and the corresponding outcome volume of the resulting sleeve.

[0229] As shown in FIG. 30, the distance (a, a 1 , 2 ) from the pylorus 4131 to the starting point for creating the sleeve. The sleeve sizes resulting from each starting point are different (e.g., for starting points 4146, 4147, 4148 at distances a, a 1 , a 2 from the pylorus 4131, they are 400 cc, 425 cc, 450 cc respectively). In one example, the control circuit prompts the user for a size selection input and, in response, presents a treatment layout plan that can be in the form of a resection path and results in the selected sleeve size. In another example, as shown in FIG. 30, the control circuit presents a plurality of resection paths 4136, 4137, 4138 and the corresponding sleeve sizes. The user can then select one of the proposed resection paths 4136, 4137, 4138, and in response, the control circuit removes the unselected resection paths.

[0230] In yet another example, the control circuit enables the user to adjust the proposed resection path on a screen showing the resection path overlaid on the virtual 3D structure and / or the operative field. The control circuit can calculate the sleeve size based on the adjustment. Alternatively, in another example, the user can select a starting point for forming the sleeve at a desired distance from the pylorus 4131. In response, the control circuit calculates the sleeve size based on the selected starting point.

[0231] For example, in a visualization system, the presentation of an excision path can be achieved by superimposing the excision path on a virtual 3D structure view and / or an operative field view. Conversely, the removal of a proposed excision path can be achieved by removing such an overlay of the excision path from the virtual 3D structure view and / or the operative field view in the visualization system.

[0232] Referring further to FIG. 30, in a particular embodiment, when the end effector of a surgical stapler clamps gastric tissue between a starting point selected from the proposed starting points 4146, 4147, 4147 and an end position 4140 at a predetermined distance from the notch angle 4132, the control circuit presents information regarding the clamp of the surgical stapler and / or information regarding the firing. In at least one example, as shown in FIG. 24, a composite data set 4012 can be displayed from visualization data 4010 and instrument data 4011. Additionally, or alternatively, values of FTC and / or FTF can be displayed. For example, the current value of FTC represented by circle 4020 is shown in real time with respect to gauge 4021, and indicator 4022 can represent the best practice FTC. Similarly, the current value of FTF represented by circle 4023 is shown with respect to gauge 4024, and indicator 4025 can represent the best practice FTF.

[0233] After the surgical stapler is fired, recommendations regarding the selection of a new cartridge can be presented on the screen of the surgical stapler or on any of the screens of the visualization system, as described in more detail below. When the surgical stapler is removed from the abdominal cavity, reloaded with the selected staple cartridge, and reintroduced into the abdominal cavity, a distance indicator that identifies a constant distance (d) from a plurality of points along the lesser curvature 4134 of the stomach 4110 to the resection path selected is overlaid on the virtual 3D structure view and / or the operative field view. To ensure the proper orientation of the end effector of the surgical stapler, the distance of the distal end of the end effector of the surgical stapler from the target, as well as the distance from the proximal end to the previously fired staple line, are overlaid on the virtual 3D structure view and / or the operative field view. This process is repeated until the resection is complete.

[0234] One or more of the distances proposed and / or calculated by the control circuit can be determined based on stored data. In at least one example, the stored data includes preoperative data, user preference data, and / or data from previous surgical procedures performed by the user or other users.

[0235] Referring to FIG. 31, process 4150 shows a control program or logic configuration for proposing a resection path for removing a portion of an anatomical organ according to at least one aspect of the present disclosure. Process 4150, as described in more detail elsewhere in this specification in relation to process 4100, at 4151 identifies the anatomical organ targeted for the surgical procedure, at 4152 identifies the anatomical structure of the anatomical organ associated with the surgical procedure, and at 4153 proposes a resection path for removing a portion of the anatomical organ by a surgical instrument. The surgical resection path is determined based on the anatomical structure. In at least one example, the surgical resection path includes different starting points.

[0236] One or more aspects of process 4150 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4150 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4150. Additionally, or alternatively, one or more aspects of process 4150 can be performed by combinational logic circuits (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuits (e.g., control circuit 420 of FIG. 2C). Further, process 4150 can be performed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0237] Referring to FIGS. 32A - 32D, a control circuit (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4100 or process 4150 may utilize dynamic visualization data to update or modify, in real - time, the layout plan of a surgical procedure during implementation. In at least one example, the control circuit modifies a set resection path (FIG. 32B) for removing a portion of an organ or an abnormality (e.g., a tumor or site) by a surgical instrument to an alternative resection path (FIG. 32D) based on dynamic visualization data from one or more imaging devices of a visualization system (e.g., visualization systems 100, 160, 500, 2108) that tracks the progress of the tissue being resected and surrounding tissue. The modification of the resection path can be triggered by a shift in the position of a critical structure (e.g., a blood vessel) relative to the resection path. For example, the tissue resection process may sometimes lead to inflammation of the tissue that changes the shape and / or volume of the tissue, thereby potentially causing a shift in the position of a critical structure (e.g., a blood vessel). The dynamic visualization data enables the control circuit to detect changes in the position and / or volume of critical structures and / or associated anatomical structures near the set resection path. If a change in position and / or volume causes a critical structure to shift into the resection path or within a safe margin from the resection path, the control circuit modifies the set resection path by selecting or at least recommending an alternative resection path for the surgical instrument.

[0238] Figure 32A shows a live view 4201 of the operative field on the screen 4230 of a visualization system. A surgical instrument 4200 is introduced into the operative field and removes a target region 4203. As shown in the enlarged view of region 4203 in Figure 32B, an initial planned layout 4209 for removing the region is overlaid on the live view 4201. Region 4203 is surrounded by critical structures 4205, 4206, 4207, 4208. As shown in Figure 32B, the initial planned layout 4209 extends a resection path around region 4203 that is within a predefined safety margin from region 4203. The resection path avoids crossing or passing over critical structures by extending either outside (e.g., critical structure 4208) or inside (e.g., critical structure 4206) of the critical structures. As described above, the initial planned layout 4209 is determined by a control circuit based on visualization data from the visualization system.

[0239] Figure 32A shows a live view 4201' of the operative field on the screen 4230 of a visualization system at a later time (00:43). The end effector 4202 of the surgical instrument 4200 excises tissue along a predefined resection path defined by the layout plan 4209. Due to a volume change of the tissue including region 4203 caused by tissue inflammation, for example, critical structures 4206 and 4208 shift into the predefined resection path. In response, as shown in Figure 32D, the control circuit proposes an alternative resection path 4210 that navigates around critical structures 4206, 4208 to protect the critical structures 4206, 4208 from damage. In various examples, an alternative resection path can be proposed to optimize the amount of healthy tissue that should remain, and to provide guidance to the user to ensure that critical structures are not hit, thereby minimizing bleeding, shortening the surgical time, reducing the pressure to handle unexpected situations, and balancing the impact on the volume of the remaining organs.

[0240] In various aspects, a control circuit that executes one or more aspects of one or more processes described by this disclosure can receive and / or derive visualization data from a plurality of imaging devices of a visualization system. The visualization data facilitates tracking of important structures outside the live view of the surgical field. A common landmark can enable the control circuit to integrate visualization data from the plurality of imaging devices of the visualization system. In at least one example, for instance, secondary tracking of important structures outside the live view of the surgical field can be achieved through a secondary imaging source, calculation of the movement of the scope, or a pre-established beacon / landmark measured by a second system.

[0241] Referring generally to FIGS. 33-35, the logical flow diagram of process 4300 illustrates a control program or logical configuration for presenting or overlaying parameters of a surgical instrument to or near a proposed surgical resection path, according to at least one aspect of this disclosure. Process 4300 is generally executed during a surgical procedure and includes, at 4301, identifying the anatomical organ targeted by the surgical procedure, and at 4302, identifying anatomical structures related to the surgical procedure from visualization data from at least one imaging device, and at 4303, proposing a surgical resection path for removing a portion of the anatomical organ by a surgical instrument. In at least one example, the surgical resection path is determined based on the anatomical structures. Process 4300 further includes, at 4304, presenting parameters of the surgical instrument according to the surgical resection path. Additionally, or alternatively, process 4300 further includes, at 4305, adjusting parameters of the surgical instrument according to the surgical resection path.

[0242] One or more aspects of process 4300 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4300 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4030. Additionally, or alternatively, one or more aspects of process 4300 can be performed by combinational logic circuitry (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuitry (e.g., control circuit 420 of FIG. 2C). Further, process 4300 can be performed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0243] In various examples, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4300, as described elsewhere herein in connection with processes 4150 (FIG. 31), 4100 (FIG. 29), at 4301 identifies an anatomical organ targeted for surgery, at 4302 identifies an anatomical structure related to the surgery from visualization data from at least one imaging device of a visualization system (e.g., visualization systems 100, 160, 500, 2108), and / or at 4303 proposes a surgical resection path for removing a portion of the anatomical organ by a surgical instrument (e.g., surgical instrument 4600). Further, a control circuit that executes one or more aspects of process 4300 can propose, or recommend, one or more parameters of the surgical instrument according to the surgical resection path proposed at 4303. In at least one example, the control circuit presents the recommended parameters of the surgical instrument at 4304 by overlaying such parameters on, or near, the proposed surgical path, as shown in FIGS. 34 and 35.

[0244] FIG. 34 shows a virtual 3D structure 4130 of the stomach of a patient undergoing sleeve gastrectomy performed using surgical instrument 4600, according to at least one aspect of the present disclosure. As described in more detail in connection with FIG. 28, for example, various elements of a visualization system (e.g., visualization systems 100, 160, 500, 2108), such as structured light projector 706 and camera 720, can be used to create visualization data to generate virtual 3D structure 4130. Related anatomical structures (e.g., pylorus 4131, angular incisure 4132, His angle 4121) are identified from visualization data from one or more imaging devices of the visualization system. In at least one example, landmarks are assigned to positions 4113, 4117, 4119 of such anatomical structures by overlaying the landmarks on virtual 3D structure 4130.

[0245] In addition, at 4303, a surgical resection path 4312 is proposed based on the identified anatomical structure. In at least one example, as shown in FIG. 34, the control circuit superimposes the surgical resection path 4312 onto the virtual 3D structure 4130. As described in more detail elsewhere herein, the proposed surgical path can be automatically adjusted based on the desired volume output. In addition, it can be automatically adjusted to match the projected margin based on the critical structures and / or tissue abnormalities automatically identified by the control circuit from the visualization data.

[0246] In various aspects, a control circuit that executes at least one aspect of process 4300 presents the parameters 4314 of a surgical instrument selected according to the surgical resection path 4312 proposed at 4303. In the example shown in FIG. 34, the parameter 4314 represents a staple cartridge automatically selected for use with the surgical instrument 4600 when performing a sleeve gastrectomy based on the surgical resection path proposed at 4303. In at least one example, the parameter 4314 includes at least one of the size of the staple cartridge, the color of the staple cartridge, the type of staple cartridge, and the staple cartridge length. In at least one example, the control circuit presents the recommended parameters 4314 of the surgical instrument 4600 at 4304 by superimposing such parameters onto or near the surgical path 4312 proposed at 4303, as shown in FIGS. 34 and 35.

[0247] In various aspects, a control circuit that executes at least one aspect of process 4300 presents tissue parameter 4315 along one or more portions of surgical resection path 4312. In the example shown in FIG. 34, tissue parameter 4315 is the tissue thickness presented by displaying a cross-section along line A-A representing the tissue thickness along at least a portion of surgical resection path 4312. In various aspects, the staple cartridge utilized by surgical instrument 4600 can be selected according to tissue parameter 4315. For example, as shown in FIG. 34, the black cartridge with larger staple size is selected for use with the thicker muscular tissue of the vestibule, and the green cartridge with smaller staple size is selected for use with the myocardial tissue of the body and the fundus of the stomach.

[0248] Tissue parameter 4315 includes at least one of tissue thickness, tissue type, and sleeve volume outcome resulting from the proposed surgical resection path 4312. Tissue parameter 4315 can be derived from previously acquired CT images, ultrasound images, and / or MRI images of the patient's organs, and / or from previously known average tissue thickness. In at least one example, surgical instrument 4600 is an intelligent instrument (similar to intelligent instrument 2112), and the tissue thickness and / or selected staple cartridge information is transmitted to surgical instrument 4600 to optimize the closure setting, firing setting, and / or any other suitable surgical instrument setting. In one example, as described in connection with FIGS. 17-19, the tissue thickness and / or selected staple cartridge information can be transmitted to surgical instrument 4600 from a visualization system (e.g., visualization systems 100, 160, 500, 2108) and a surgical hub (e.g., surgical hubs 2106, 2122) that communicate with surgical instrument 4600.

[0249] In various examples, a control circuit that executes at least one aspect of process 4300 proposes an arrangement 4317 of sizes (e.g., 45 mm and 60 mm) of two or more staple cartridges according to a determined tissue thickness along at least a portion of surgical resection path 4312. Further, as shown in FIG. 35, the control circuit can present the arrangement 4317 along the surgical resection path 4312 proposed at 4303. Alternatively, the control circuit can present an appropriate arrangement 4317 along a surgical resection path selected by the user. The control circuit can determine the tissue thickness along the resection path selected by the user as described above and propose a staple cartridge arrangement according to this tissue thickness.

[0250] In various aspects, a control circuit that executes one or more aspects of process 4300 can propose a surgical resection path or optimize a selected surgical resection path so as to minimize the number of staple cartridges in the staple cartridge arrangement 4317 without the size of the resulting sleeve being impaired beyond a predetermined threshold. By reducing the number of used cartridges, the treatment time and cost are reduced, and patient trauma is reduced.

[0251] Continuing to refer to FIG. 35, the arrangement 4317 includes a first staple cartridge 4352 and a final staple cartridge 4353 that define the start and end of the surgical resection path 4312. If only a small portion of the final staple cartridge 4353 of the proposed staple cartridge arrangement 4317 is required, the control circuit can adjust the surgical resection path 4312 so as to eliminate the need for the final staple cartridge 4353 without the size of the resulting sleeve being impaired beyond a predetermined threshold.

[0252] In various examples, a control circuit that executes at least one aspect of process 4300 presents a virtual firing of a proposed staple cartridge arrangement 4317 that virtually separates the virtual 3D structure 4130 into a retention portion 4318 and a removal portion 4319, as shown in FIG. 35. The retention portion 4318 is a virtual representation of the sleeve resulting from the implementation of the proposed surgical resection path 4312 by firing the staple cartridge arrangement 4317. The control circuit can further determine the estimated volume of the retention portion 4318 and / or the removal portion 4319. The volume of the retention portion 4318 represents the volume of the resulting sleeve. In at least one example, the volume of the retention portion 4318 and / or the removal portion 4319 is derived from visualization data. In another example, the volume of the retention portion 4318 and / or the removal portion 4319 is determined from a database that stores retention volume, removal portion volume, and corresponding surgical resection paths and can be constructed from previous surgeries on organs resected by the same or at least similar dimensions and at least similar resection paths.

[0253] In various examples, a combination of predetermined average tissue thickness data based on organ situation recognition, as described in more detail above, can be used to select a first staple cartridge for arrangement 4317 by combining volume analysis from a visualization source and, if available for the patient, CT, MRI, and / or ultrasonic secondary imaging. The firing of subsequent staple cartridges within arrangement 4317 can optimize the firing parameters using instrument data from previous firings in addition to the visualization data. Instrument data that can be used to supplement the volume measurement includes, for example, FTF, FTC, draw current by a motor driving firing and / or closure, closure gap of the end effector, firing speed, tissue impedance measurement across the jaw, and / or waiting or resting time during the use of the surgical instrument.

[0254] In various examples, for example, visualization data such as structured light data can be used to track changes in the surface geometry of tissue being treated by a surgical instrument (e.g., surgical instrument 4600). Further, for example, visualization data such as spectral data can be used to track important structures below the tissue surface. Structured data and / or spectral data can be used to maintain contact between a set instrument and tissue throughout tissue treatment.

[0255] In at least one example, the end effector 4642 of the surgical instrument 4600 can be used to grasp tissue between the jaws. For example, once the desired tissue-to-instrument contact is confirmed by user input, visualization data of the end effector and surrounding tissue associated with the desired tissue-to-instrument contact can be used to automatically maintain the desired tissue-to-surface contact throughout at least a portion of the tissue treatment. The desired tissue-to-surface contact can be automatically maintained, for example, by slight manipulation of the position, orientation, and / or FTC parameters of the end effector 4642.

[0256] If the surgical instrument 4600 is a handheld surgical instrument, for example, the user can be provided with position and / or orientation manipulation in the form of instructions that can be presented on a display 4625 (FIG. 22) of the surgical instrument 4600. Also, if a user operation is required to re-establish the desired tissue-to-surface contact, the surgical instrument 4600 can issue an alert. On the other hand, non-user operations, for example, operations on FTC parameters and / or joint angles, can communicate from, for example, the surgical hub 2106 or the visualization system 2108 to the controller 4620 of the surgical instrument 4600. The controller 4620 can then cause the motor driver 4626 to implement the desired operation. If the surgical instrument 4600 is a surgical instrument coupled to a robotic arm of the robotic system 2110, position and / or orientation manipulation can communicate from, for example, the surgical hub 2106, or the visualization system 2108 to the robotic system 2110.

[0257] Referring mainly to FIGS. 36A - 36C, the firing of a surgical instrument 4600 loaded with a first staple cartridge 4652 into a staple cartridge arrangement 4317 is illustrated. In a first step, as shown in FIG. 36A, a first landmark 4361 and a second landmark 4362 are superimposed on a surgical resection path 4312. The landmarks 4361, 4362 are separated by a distance (d1) defined by the size (e.g., 45) of the staple cartridge 4652, which represents the length of a staple line 4363 to be deployed on the surgical resection path 4312 by the staple cartridge 4652. A control circuit that executes one or more aspects of the process 4300 employs visualization data as described in more detail elsewhere in this specification to superimpose the landmarks 4361, 4362 on the surgical resection path 4312 and, for example, continuously track and update their positions relative to certain predefined critical structures such as anatomical structures 4364, 4365, 4366, 4367, etc.

[0258] As shown in FIG. 36B, during firing, the staples of the staple line 4363 are deployed into the tissue, and a cutting member 4645 advances along the surgical resection path 4312 between the landmarks 4361 and 4362 to cut the tissue. In various examples, as the cutting member 4645 advances, the tissue being treated stretches and / or shifts. Tissue stretch and / or shift beyond a predetermined threshold indicates that the movement of the cutting member 4645 through the tissue being treated is too fast.

[0259] FIG. 37 is a logic flow diagram of a process 4170 showing a control program, or logical configuration, for adjusting the firing speed of a surgical instrument to account for tissue stretch and / or shift during firing. The process 4170 includes, at 4171, monitoring tissue stretch / shift during firing of the surgical instrument, and at 4172, if the tissue stretch / shift is above a predetermined threshold, at 4173, adjusting the firing parameters.

[0260] One or more aspects of process 4170 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4170 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4170. Additionally, or alternatively, one or more aspects of process 4170 can be performed by combinational logic circuitry (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuitry (e.g., control circuit 420 of FIG. 2C). Further, process 4170 can be performed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0261] In various examples, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4170, at 4171, uses visualization data from a visualization system (e.g., visualization systems 100, 160, 500, 2108) to monitor tissue stretch / shift during the firing of surgical instrument 4600. In the example shown in FIG. 36B, at 4171, the visualization data is used to track the distortion of a structured light grid projected onto the tissue during firing and / or to track landmarks 4364, 4365, 4366, 4367 that represent the positions of adjacent anatomical structures, thereby monitoring tissue stretch / shift (d). Additionally, or alternatively, at 4171, tissue stretch (d) can be monitored by tracking the position of landmark 4362 during firing. In the example of FIG. 36B, tissue stretch / shift (d) is the difference between the distance (d1) between landmarks 4361 and 4362 during firing and the distance (d2) between landmarks 4361 and 4362 during firing. In any case, at 4172, if the tissue stretch / shift (d) is greater than or equal to a predetermined threshold, at 4173, the control circuit adjusts the firing parameters of surgical instrument 4600 to reduce the tissue stretch / shift (d). For example, the control circuit can cause controller 4620 to reduce the speed of firing motor drive assembly 4604 by, for example, reducing the pull-in current of firing motor 4602 and reducing the forward speed of cutting member 4645. Additionally, or alternatively, the control circuit can cause controller 4620 to pause firing motor 4602 for a predetermined period to reduce the tissue stretch / shift (d).

[0262] As shown in FIG. 36C, after firing, the jaws of end effector 4642 are unclamped and the stapled tissue contracts due to the fired staples in staple line 4363. FIG. 36C shows the projected staple line length defined by distance (d1) and the actual staple line defined by a distance (d3) that is less than distance (d1). The difference between distances d1 and d2 represents the contraction / shifting distance (d’).

[0263] FIG. 38 is a logical flow diagram of a process 4180 showing a control program or logical configuration for adjusting a proposed staple cartridge placement along a proposed surgical resection path. Process 4180, at 4081, after firing a staple cartridge of the proposed placement, monitors shrinkage / misalignment of the stapled tissue along the proposed surgical resection path and includes adjusting a proposed subsequent staple cartridge position along the proposed surgical resection path.

[0264] One or more aspects of process 4180 can be executed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4180 are executed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to execute one or more aspects of process 4180. Additionally, or alternatively, one or more aspects of process 4180 can be executed by combinational logic circuits (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuits (e.g., control circuit 420 of FIG. 2C). Further, process 4180 can be executed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0265] In various examples, a control circuit (e.g., block control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) that executes one or more aspects of process 4180 monitors, at 4181, the shrinkage / misalignment of stapled tissue along the proposed resection path 4312. In the example shown in FIG. 36C, staple line 4363 is deployed from the staple cartridge of staple cartridge arrangement 4317 into the tissue between landmarks 4361 and 4362. When the jaws of end effector 4642 are unclamped, the stapled tissue shrinks / misaligns by a distance (d’). The distance (d’) is the difference between the pre-firing distance (d1) between landmarks 4361 and 4362 that represents the length of the proposed staple line 4361 as proposed by arrangement 4317, and the distance (d3) that represents the actual length of staple line 4363.

[0266] To avoid gaps between successive staple lines, the control circuit adjusts the subsequent staple cartridge positions of arrangement 4317 along the proposed surgical resection path 4312. For example, as shown in FIG. 36C, the initially proposed staple line 4368 is removed and replaced by an updated staple line 4369 that extends through or covers the gap defined by distance (d’). In various aspects, at 4181, the shrinkage / misalignment (d’) of the tissue is monitored by using visualization data to track the distortion of a structured light grid projected onto the tissue after the jaws of end effector 4642 are unclamped, and / or by tracking landmarks 4364, 4365, 4366, 4367 that represent the positions of adjacent anatomical structures. Additionally, or alternatively, at 4181, the shrinkage distance (d’) of the tissue can be monitored by tracking the position of landmark 4362.

[0267] In various aspects, it may be desirable to use non-visualization data from a non-visualization system to corroborate visualization data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108), and vice versa. In one example, the non-visualization system can include a ventilator configured to measure non-visual data such as the volume, pressure, partial pressure of carbon dioxide (PCO 2 ), and partial pressure of oxygen (PO 2 ) of a patient's lungs. By corroborating visualization data with non-visualization data, a clinician can more confidently provide that the visualization data derived from the visualization system is accurate. Additionally, as will be described in detail below, by corroborating visualization data with non-visualization data, a clinician can identify postoperative complications and determine the overall efficiency of an organ. Corroboration can also be useful in the case of segmentectomy or complex lobectomy without a fissure.

[0268] In various aspects, a clinician may need to remove a portion of a patient's organ to remove important structures such as tumors and / or other tissues. In one example, the patient's organ can be the right lung. A clinician may need to remove a portion of the patient's right lung to remove diseased tissue. However, a clinician may not want to remove too much of the patient's lung during surgery to ensure that the lung function is not overly impaired. Lung function can be evaluated based on the maximum lung volume per breath, which represents the vital capacity. When determining how much of the lung can be safely removed, a clinician is limited by a predetermined reduction in the maximum lung volume, beyond which the lung loses its viability and a complete organ resection is required.

[0269] In at least one example, the surface area and / or volume of the lung is estimated from visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). The surface area and / or volume of the lung can be estimated at the maximal inspiratory capacity or the maximal lung volume per breath. In at least one example, the surface area and / or volume of the lung can be estimated at multiple points throughout the inhalation / exhalation cycle. In at least one aspect, prior to excising a portion of the lung, visualization data and non-visualization data can be utilized to correlate the surface area and / or volume of the lung determined by the visualization system with the lung volume determined by a ventilator. The correlation data can be employed, for example, in constructing a mathematical relationship between the surface area and / or volume of the lung, such as derived from the visualization data, and the lung volume determined by the ventilator. This relationship can be employed in estimating the size of the lung portion that can be removed while maintaining a reduction in maximal lung volume below a predetermined threshold value that maintains the viability of the lung.

[0270] FIG. 39 shows a logical flow diagram of a process 4750 for suggesting a surgical resection of a portion of an organ, according to at least one aspect of the present disclosure. Process 4750 is generally performed during a surgical procedure. Process 4750 includes, at 4752, suggesting a portion of the organ to be resected based on visualization data from a surgical visualization system, the resection of the portion being configured to result in an estimated volume reduction of the organ. Process 4750 can further include, at 4754, determining a first value of a non-visualization parameter of the organ prior to resection of the portion, and, at 4756, determining a second value of the non-visualization parameter of the organ after resection of the portion. Further, in a particular example, process 4750 can further include, at 4758, corroborating a predetermined volume reduction based on the first value and the second value of the non-visualization parameter.

[0271] One or more aspects of process 4750 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4750 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4750. Additionally, or alternatively, one or more aspects of process 4750 can be performed by combinational logic circuitry (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuitry (e.g., control circuit 420 of FIG. 2C). Further, one or more aspects of process 4750 can be performed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0272] In various aspects, process 4750 can be implemented by a computer-implemented interactive surgical system 2100 (FIG. 19) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that can include a remote server 2113. A control circuit that performs one or more aspects of process 4750 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0273] Figure 41A shows a set of lungs 4780 of a patient. In one embodiment, a clinician can utilize an imaging device 4782 to project a light pattern 4785, such as stripes, grid lines, and / or dots, onto the surface of the patient's right lung 4786 to enable determination of the topography or landscape of the surface of the patient's right lung 4786. The imaging device can be similar to the imaging device 120 (FIG. 1) at various points. As described elsewhere herein, a projected light array can be employed to determine the shape defined by the surface of the patient's right lung 4786 and / or the movement of the patient's right lung 4786 during the procedure. In one embodiment, the imaging device 4782 can be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system, such as the surgical visualization system 100, can utilize the surface mapping logic 136 of the control circuit 133 to determine the topography or landscape of the surface of the patient's right lung 4786, as described elsewhere herein.

[0274] A clinician can provide a surgical system, such as the surgical system 2100, with the type of procedure to be performed, such as a right upper lobectomy. In addition to providing the surgical procedure to be performed, the clinician can provide the surgical system with the maximum desired volume of the organ to be removed during the surgery. Based on the visualization data obtained from the imaging device 4782, the type of surgical procedure to be performed, and the maximum desired volume to be removed, the surgical system can propose an excision path 4788 for removing a portion 4790 of the right lung that satisfies all of the clinician's inputs. Other methods of proposing a surgical excision path are described elsewhere herein. To propose the excision path 4788, the surgical system can consider any number of additional parameters.

[0275] In various examples, it may be desirable to ensure that the desired volume reduction from the patient's organ is achieved based on the volume of the resected organ. Non-visualization data from a non-visualization system can be utilized to verify that the resected volume results in the desired volume reduction. In one embodiment, a ventilator can be used to measure the patient's maximal lung volume over time.

[0276] In at least one example, a clinician can use the surgical system 2100 in a surgical procedure to remove a lung tumor. The control circuit can identify the tumor from visualization data as described above in connection with FIGS. 13A-13E, and can propose a surgical resection path to provide a safety margin around the tumor as described above in connection with FIGS. 29-38. The control circuit can further estimate the volume of the lung at maximal lung volume. A ventilator can be used to measure the maximal lung volume prior to the surgical procedure. The control circuit can use a predetermined mathematical correlation between the visually estimated lung volume and lung capacity detected by the ventilator to estimate the maximal lung volume reduction associated with removing a portion of the lung that includes the tumor and its surrounding tissue. If the estimated lung capacity reduction exceeds a predetermined safety threshold, the control circuit can warn the clinician and / or propose an alternative surgical resection path that results in a lower lung capacity.

[0277] FIG. 41C shows a graph 4800 that measures the patient's maximal lung volume over time. Prior to resection of a portion of the organ (at time t 1 ), the ventilator can measure the maximal lung volume. In FIG. 41C, at time t prior to resection of portion 4790 1In this case, the maximum lung volume is measured to be 6 L. In the above example where the surgical procedure to be performed is a right upper lobectomy, the clinician may wish to remove only the volume of the patient's lung that results in a predetermined volume reduction so as not to impair the patient's respiratory power. In one embodiment, the clinician may wish to remove, for example, a portion that reduces the patient's maximum lung volume by up to about 17% at most. Based on the surgical procedure and the desired volume reduction, the surgical system can propose a surgical resection path 4788 that achieves the removal of a portion of the lung while maintaining the maximum lung volume at a value of 83% or more of the maximum lung volume that remains unexcised.

[0278] As shown in FIG. 41C, the clinician can use ventilator data to monitor the patient's maximum lung volume over time, such as before 4802 and after 4804 the resection of a portion 4790 of the lung. At time t 2 At this time, a portion 4790 of the lung is resected along the proposed resection path 4788. As a result, the maximum lung volume measured by the ventilator decreases. The clinician can verify with ventilator data (the maximum lung volume before resection 4802 and the maximum lung volume after resection 4804) to ensure that the volume of the lung after resection results in the desired volume reduction from the lung. As shown in FIG. 41C, after resection, the maximum lung volume decreases to 5 L, which is a decrease in the maximum lung volume of about 17% and is approximately the same as the desired volume reduction. The clinician can have a stronger conviction that the actual volume reduction matches the desired volume reduction achieved by the proposed surgical resection path 4788 using the ventilator data. In other embodiments, if there is a discrepancy between the non-visualized data and the visualized data, such as the decrease in the maximum lung volume being greater than expected (too much lung resection) or the decrease in the maximum lung volume being less than expected (insufficient lung resection), the clinician can determine whether appropriate actions are necessary.

[0279] Referring now to FIG. 41B, the right lung 4792 of a patient after resection of a portion 4790 is shown. After resection of the portion 4790, a clinician may inadvertently cause an air leak 4794, which results in air leaking into the space between the lung 4794 and the chest wall, potentially causing a pneumothorax 4796. As a result of the air leak 4794, the patient's maximum lung volume per breath steadily decreases over time as the right lung 4792 collapses. By using visualization data derived from a visualization system (e.g., visualization systems 100, 160, 500, 2108) to perform a dynamic surface area / volume analysis of the lung and visually tracking the volume changes of the lung, an air leak can be detected. The volume and / or surface area of the lung is visually tracked at one or more points during the inhalation / exhalation cycle to detect volume changes indicative of an air leak 4794. In one embodiment, as previously discussed, an array of projection light from the imaging device 4782 is employed to monitor the movement of the patient's right lung 4786 over time, such as monitoring for size reduction. In another embodiment, surface mapping logic, such as a surgical visualization system, surface mapping logic 136, is utilized to determine the topography or landscape of the surface of the patient's right lung 4786 and monitor for changes in the topography or landscape over time.

[0280] In one aspect, a clinician can use a non-visualization system, such as a ventilator, to corroborate the volume reduction detected by the visualization system. Referring again to FIG. 41C, as described above, at 4802, the patient's maximum lung volume is measured before and after resection of a portion of the lung to corroborate that the desired lung volume reduction matches the actual lung volume reduction. In the example above where an air leak occurs inadvertently, at 4806, the maximum lung volume can steadily decrease over time. In one example, at a time t 2 after resection of a portion, the clinician may notice that the maximum lung volume has decreased from 6 L to 5 L, which roughly matches the desired decrease in lung volume. After resection of a portion, the surgical visualization system can monitor the patient's lung volume over time. If the surgical visualization system determines that there is a change in volume, the clinician at time t 3At this time, the maximum lung capacity can be measured again. At time t 3 At this time, the clinician may notice that the maximum lung capacity has decreased from 5L to 4L, which corroborates the data determined from the visualization system and indicates that there may be an air leak in the right lung 4792.

[0281] In addition, the control circuit can be configured to measure organ efficiency based on the visualization data and non-visualization data. In one aspect, the organ efficiency can be determined by comparing the visualization data with the difference in non-visualization data before and after a partial resection. In one example, the visualization system can generate a resection path that reduces the maximum lung capacity by 17%. The ventilator can be configured to measure the maximum lung capacity before and after a partial resection. In the example shown in Figure 41C, the maximum lung capacity has decreased by approximately 17% (from 6L to 5L). Since the decrease in the actual lung capacity (17%) relative to the desired lung capacity (17%) is almost 1:1, the clinician can determine that the lung is functionally efficient. In another example, the visualization system can generate a resection path that reduces the maximum lung capacity by 17%. However, the ventilator can measure a decrease in the maximum lung capacity that is greater than 17%, such as 25%. In this example, when a portion of the lung is resected, the maximum lung capacity decreases more than expected, so the clinician can determine that the lung is not functionally efficient.

[0282] Figure 40 shows a logic flow diagram of a process 4760 for estimating a reduction in the volume of an organ resulting from the removal of a selected portion of the organ, according to at least one aspect of the present disclosure. The process 4760 is similar to the process of Figure 4750 in many respects. However, unlike the process 4750, the process 4760 depends on the clinician selecting or proposing a surgical resection path when removing a portion of the organ during surgery. The process 4760 includes, at 4762, receiving from the user an input indicating a portion of the organ to be resected. The process 4760 further includes, at 4764, estimating the reduction in the volume of the organ by removing the portion. In at least one example, the organ is the patient's lung, and the estimated volume reduction at 4762 is the reduction in the maximum lung volume per breath of the patient's lung. Visualization data from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108) can be employed to estimate the volume reduction corresponding to the removal of the portion. The process 4760 can further include, at 4766, determining a first value of a non-visualization parameter of the organ before resection of the portion, and, at 4768, determining a second value of the non-visualization parameter of the organ after resection of the portion. Finally, the process 4760 can further include, at 4768, corroborating the estimated reduction in the volume of the organ based on the first value of the non-visualization parameter and the second value of the non-visualization parameter.

[0283] One or more aspects of process 4760 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4760 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4760. Additionally, or alternatively, one or more aspects of process 4760 can be performed by combinational logic circuitry (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuitry (e.g., control circuit 420 of FIG. 2C). Further, one or more aspects of process 4760 can be performed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0284] In various aspects, process 4760 can be implemented by a computer-implemented interactive surgical system 2100 (FIG. 19) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that can include a remote server 2113. A control circuit that performs one or more aspects of process 4760 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0285] In one example, a clinician can provide an input indicating a portion of an organ to be resected to a surgical visualization system such as surgical visualization system 2100. In one example, the clinician can draw a resection path on a virtual 3D structure of an organ, such as the virtual 3D structure generated at 4104 during process 4100. In other examples, the visualization system can overlay a treatment layout plan, which can be in the form of a recommended treatment path, more detailed elsewhere in this specification. The recommended treatment path can be based on the type of surgical procedure being performed. In one embodiment, the recommended treatment path can propose various starting points and various resection paths for the clinician to select, similar to resection paths 4146, 4147, 4148 described elsewhere in this specification. The proposed resection path can be determined by the visualization system to avoid certain critical structures such as arteries. The clinician can select the proposed resection path until the desired resection path for removing a portion of the organ is complete.

[0286] In one example, the surgical visualization system can determine an estimated volume reduction of the organ based on the selected resection path. After resected a predetermined portion along the resection path, the clinician may wish to use non-visualization data to verify that, based on the visualization data, the actual volume reduction matches the estimated volume reduction. In one embodiment, this verification can be performed using a similar procedure as described above for process 4750 where the organ is the lung. The clinician can measure the maximum lung volume before 4802 and after 4804 resection of the lung and compare the change in the maximum lung volume to determine the actual decrease in the maximum lung volume. In one example, the surgical visualization system can estimate that the maximum lung volume will decrease by 17% based on the resection path proposed by the clinician. Before resection, the clinician may notice that the maximum lung volume is 6L (time t 1 ) and after resection, the clinician may notice that the maximum lung volume is 5L (time t 2) One might notice that this would result in a decrease of about 17% in the total lung capacity. Clinicians can have a stronger conviction that the actual volume reduction is consistent with the estimated volume reduction by using this non-visualization data / ventilator data. In other cases, if there is a discrepancy between the non-visualization data and the visualization data, such as the decrease in total lung capacity being greater than expected (too much lung resection) or less than expected (insufficient lung resection), clinicians can determine whether appropriate actions are needed.

[0287] In addition, the control circuit can be configured to measure organ efficiency based on the visualization data and the non-visualization data. In one aspect, the organ efficiency can be determined by comparing the visualization data with the difference in non-visualization data before and after a partial resection. In one example, the surgical visualization system can estimate that the total lung capacity will decrease by 17% based on the desired resection path of the clinician. The ventilator can be configured to measure the total lung capacity before and after a partial resection. In the example shown in FIG. 41C, the total lung capacity has decreased by approximately 17% (from 6 L to 5 L). Since the decrease in the actual lung capacity (17%) relative to the estimated lung capacity (17%) is almost 1:1, the clinician can determine that the lung is functionally efficient. In another example, the surgical visualization system can estimate that the total lung capacity will decrease by 17% based on the desired resection path of the clinician. However, the ventilator can measure a decrease in total lung capacity that is greater than 17%, such as 25%. In this example, when a portion of the lung is resected, the total lung capacity decreases more than expected, so the clinician can determine that the lung is not functionally efficient.

[0288] As described above with respect to processes 4750, 4760, clinicians can corroborate visualization data with non-visualization data, such as by using a ventilator to measure the total lung capacity before and after resection of a portion of the lung. Another example of corroborating visualization data with non-visualization data is by capnography.

[0289] Figure 42 shows a graph 4810 that measures the partial pressure of carbon dioxide (PCO 2 ) exhaled by a patient over time. In other examples, the partial pressure of oxygen (PO 2 ) exhaled by the patient can be measured over time. Graph 4810 shows the PCO 2 measured before resection 4812, immediately after resection 4814, and 1 minute after resection 4816. In Figure 42, before resection 4812, the PCO 2 is measured to be approximately 40 mmHg (at time t 1 ). In the above example where the surgical procedure being performed is a right upper lobectomy, the visualization system can expect or estimate that the lung volume will decrease by 17%. The measured PCO 2 level by the ventilator can be used to support this desired or estimated volume reduction.

[0290] As shown in Figure 42, a clinician can use ventilator data to monitor the patient's PCO 2 over time, such as before resection 4812 and after resection 4814 of a portion 4790 of the lung. At time t 2 , a portion 4790 of the lung is resected, and as a result, at 4818, the PCO 2 measured by the ventilator can decrease. The clinician can use the ventilator data (pre-resection PCO 2 4812 (at t 1 ), and post-resection PCO 2 4814 (at t 2 ) to support that the actual decrease in lung volume matches the estimated or desired decrease in lung volume. As seen in Figure 42, immediately after resection 4814 of portion 4790, the PCO 2 decreases 4812, which can be measured as a decrease of approximately 17% of the PCO 2 (about 33.2 mmHg). The clinician can be more confident using this non-visualization data / ventilator data that the actual decrease in lung volume matches the desired or estimated decrease in lung volume.

[0291] In other examples, the clinician can use non-visualized data / PCO 2 data to determine the discrepancy when compared with visualized data. In one example, at time t 2 4814 immediately after resection of a portion 4790, PCO 2 can be measured at 4820, which is higher than the PCO 2 measured before resection at 4812. The increase in PCO 2 may be the result of the bronchus being inadvertently blocked during surgery and CO 2 accumulating within the patient. In another example, at time t 2 4814 immediately after resection of a portion 4790, PCO 2 can be measured at 4822, which is lower than the PCO 2 measured before resection at 4812 and may be measured lower than expected. The decrease in PCO 2 may be due to a blood vessel being inadvertently blocked during surgery, resulting in less O 2 being delivered to the body and, as a result, less CO 2 being produced. In either case, the clinician can take appropriate action to remedy the situation.

[0292] Also, changes in PCO 2 can be measured at times other than immediately after resection at 4814, such as at 4816 (time t 3 , etc.) one minute after resection. At time t 3 , other body functions (such as the liver) compensate for changes in PCO 2 as a result of the resection. In this situation, PCO 2 can be measured at approximately 40 mmHg, i.e., roughly the same as before resection at 4812. At time t 3 , the difference measured between the PCO 2 before resection at 4812 may indicate the aforementioned inadvertent blockage. For example, at time t 3 , PCO 2 is measured at 4824 to be higher than before resection at 4812, indicating the possibility that the bronchus was inadvertently blocked, or PCO 2is 4826, which is measured lower than 4812 before resection and may indicate that the blood vessel was inadvertently occluded.

[0293] In addition, the control circuit can be configured to measure organ efficiency based on visualization data and non-visualization data. In one aspect, the organ efficiency can be determined by comparing the visualization data with the difference in non-visualization data before and after a portion of the resection. In one example, the surgical visualization system may estimate that the lung volume decreases by 17% based on the desired resection path of the clinician. The ventilator can be configured to measure PCO 2 before a portion of the resection and after a portion of the resection. In the embodiment shown in FIG. 42, immediately after resection 4814, PCO 2 decreases by approximately 17%. Since the (17%) decrease in PCO 2 for the estimated (17%) decrease in lung volume is almost 1:1, the clinician can determine that the lung is functionally efficient. In another example, the surgical visualization system may estimate that the lung volume decreases by 17% based on the desired resection path of the clinician. However, the ventilator can measure a decrease in PCO 2 greater than 17%, such as 25%. In this example, when a portion of the lung is resected, PCO 2 decreases more than expected, so the clinician can determine that the lung is not functionally efficient.

[0294] In addition to the above maximum lung volume and maximum PCO 2 , non-visualization parameters including blood pressure or EKG data can be utilized. The EKG data provides approximate frequency data regarding the deformation of the artery. This frequency data, within a similar frequency range where the surface geometry changes, can help identify important vascular structures.

[0295] As described above, it may be desirable to utilize non-visualization data from a non-visualization system to corroborate visualization data derived from a surgical visualization system (e.g., visualization systems 100, 160, 500, 2108). In the above example, the non-visualization data is a means for corroborating visualization data after a portion of an organ has already been excised. In this example, it may be desirable to supplement visualization data with non-visualization data before a portion of the organ is excised. In one example, the non-visualization data can be used together with the visualization data to help determine the characteristics of the organ being operated on. In one aspect, this characteristic can be an abnormality of the organ tissue that may not be suitable for cutting. The non-visualization data and the visualization data can help inform the surgical visualization system and the clinician about areas to avoid when planning the excision path of the organ. This can also be useful in cases of segmentectomy or complex lobectomy without a fissure.

[0296] FIG. 43 shows a logical flow diagram of a process 4850 for detecting tissue abnormalities based on visualization data and non-visualization data, according to at least one aspect of the present disclosure. Process 4850 is generally performed during a surgical procedure. Process 4850 can include, at 4852, receiving, from a surgical visualization system, first visualization data of an organ in a first state, and, at 4854, determining a first value of a non-visualization parameter of the organ in the first state. Further, process 4850 can include, at 4856, receiving, from the surgical visualization system, second visualization data of the organ in a second state, and, at 4858, determining a second value of the non-visualization parameter of the organ in the second state. Also, the process can include, at 4860, detecting a tissue abnormality based on the first visualization data, the second visualization data, the first value of the non-visualization parameter, and the second value of the non-visualization parameter.

[0297] One or more aspects of process 4850 can be performed by one or more of the control circuits (e.g., control circuits 132, 400, 410, 420, 602, 622, 2108, 4620) described by this disclosure. In at least one example, one or more aspects of process 4850 are performed by a control circuit (e.g., control circuit 400 of FIG. 2A), which circuit includes a processor and a memory storing a set of computer-executable instructions that, when executed by the processor, cause the processor to perform one or more aspects of process 4850. Additionally, or alternatively, one or more aspects of process 4850 can be performed by combinational logic circuitry (e.g., control circuit 410 of FIG. 2B) and / or sequential logic circuitry (e.g., control circuit 420 of FIG. 2C). Further, one or more aspects of process 4850 can be performed by any suitable circuit having any suitable hardware and / or software components that can be disposed within or associated with various suitable systems described by this disclosure.

[0298] In various aspects, process 4850 can be implemented by a computer-implemented interactive surgical system 2100 (FIG. 19) that includes one or more surgical systems 2102 and a cloud-based system (e.g., cloud 2104 that can include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with cloud 2104, which can include remote server 2113. The control circuit that performs one or more aspects of process 4850 can be a component of a visualization system (e.g., visualization systems 100, 160, 500, 2108).

[0299] Figure 44A shows the right lung 4870 of a patient in a first state 4862. In one example, the first state 4862 can be a contracted state. In another example, the first state 4862 can be a collapsed state. An imaging device 4872 inserted through a cavity 4874 in the patient's chest wall 4876 is shown. A clinician can use the imaging device 4872 to project a light pattern 4882, such as stripes, grid lines, and / or dots, onto the surface of the right lung 4870 at 4880 to enable determination of the topography or landscape of the surface of the patient's right lung 4870. The imaging device can be similar to the imaging device 120 (FIG. 1) in various respects. As described elsewhere herein, a projected light array is employed to determine the shape defined by the surface of the patient's right lung 4870 and / or the movement of the patient's right lung 4870 during the procedure. In one embodiment, the imaging device 4782 can be coupled to the structured light source 152 of the control system 133. In one embodiment, a surgical visualization system, such as the surgical visualization system 100 described elsewhere herein, can utilize the surface mapping logic 136 of the control circuit 133 to determine the topography or landscape of the surface of the patient's right lung 4786. In the first state 4862 of the right lung 4870, a ventilator is used to measure parameters of the right lung 4870, such as the pressure (P 1 , or positive end-expiratory pressure (PEEP)) in the first state, or the volume (V 1 ) of the right lung 4870 in the first state.

[0300] Figure 44B shows the right lung 4870 of a patient in a second state 4864. In one example, the second state 4864 can be a partially inflated state. In another example, the second state 4864 can be a fully inflated state. The imaging device 4872 can be configured to continuously project the light pattern 4882 onto the surface of the lung 4870 at 4880 to enable determination of the topography or landscape of the surface of the patient's right lung 4870 in the second state 4864. In the second state 4864 of the right lung 4870, a ventilator is used to measure a pressure (P 1 ) in the second state that is greater than the pressure P 2 in the first state 4862, and the volume V 1Volume of the second state that is larger than (V 2 ) etc., parameters of the right lung 4870 can be measured.

[0301] Based on the surface topography determined from the surgical visualization system and the imaging device 4872, together with the non-visualized data (pressure / volume) determined from the ventilator, the surgical visualization system can be configured to determine abnormalities in the tissue of the right lung 4870. In one example, in the first state 4862, the imaging device 4872 can determine the topography of the first state 4662 of the right lung 4870 (shown in FIG. 44A and shown in more detail in FIG. 44C), and the ventilator can determine the pressure / volume of the first state. In the second state 4864, the imaging device 4872 can determine the topography of the second state 4864 of the right lung 4870 (shown in FIG. 44D and shown in more detail by FIG. 44D), and the ventilator can determine the pressure / volume of the second state that is larger than the pressure / volume of the first state due to the lung being partially or fully inflated. The visualization system can be configured to monitor changes in the topography of the right lung 4870 according to a known increase in pressure / volume based on the known increase in pressure / volume. In one aspect, this pressure / volume measurement value from the ventilator can be correlated with the surface deformation of the right lung 4870 to identify diseased areas in the lung and be useful for staple placement.

[0302] In one aspect, referring to FIGS. 44B and 44D, when the pressure increases from P 1 to P 2 (the volume increases from V 1 to V 2When increased (to), the surface topography determined from structured light 4880 has changed compared to the first state 4862. In one example, the pattern of light 4882 can be dots, and the dots are spaced apart from each other by a distance(s) as the lung size increases. In another example, the pattern of light 4882 can be grid lines, and the grid lines are spaced apart or contoured as the lung size increases. The imaging device can determine an area 4886 that did not change according to a known pressure and volume increase based on the known pressure increase. For example, when the imaging device 4872 projects a pattern 4882 of grid lines and dots onto the surface of the right lung 4870 (shown in FIGS. 6A-6D), the visualization system can be configured to monitor the contours of the grid lines and the relative positions of the dots with respect to the known pressure / volume increase. When the visualization system notifies of irregularities in the dot spacing or the position and curvature of the grid lines, the visualization system can determine that these areas correspond to areas where there is a possibility of tissue abnormalities such as sub-surface voids 4886, or areas where important structures 4884 such as tumors can be located. In one embodiment, referring to process 4100 that identifies the anatomical structure of at least a portion 4105 of an anatomical organ related to a surgical procedure, process 4100 can identify abnormalities as described above herein and overlay these abnormalities onto a 3D structure.

[0303] In one example, the patient may be suffering from emphysema, a lung disease that causes shortness of breath. In people with emphysema, the air sacs (alveoli) in the lungs are damaged, and over time the inner walls of the air sacs become weak and rupture, creating large air cavities instead of many small air sacs. This results in O 2 / CO 2The internal surface area of the lungs used for gas exchange is reduced, and thus the amount of oxygen reaching the bloodstream is reduced. In addition, damaged alveoli do not function properly, old air is trapped, and there is no room for fresh air rich in oxygen to enter. The air spaces in the lungs of patients with emphysema represent areas with thin tissue thickness, and the outcome of stapling in those areas may be affected. When the tissue also becomes weak, as a result, the alveoli rupture and the staples passing through them can no longer hold well.

[0304] When the lungs with emphysema expand and contract, the areas with sub-surface air spaces deform differently due to pressure changes compared to healthy tissue. By using the above-mentioned process 4850, clinicians can be informed to detect these weak tissue areas containing sub-surface air spaces and avoid stapling through these areas, thereby reducing the possibility of postoperative air leakage. Due to the tissue deformation force of this process 4850, these differences can be detected, and surgeons can be guided to place the stapler.

[0305] In the second example, the patient may have tissue separation due to cancer. Before treatment, the tumor may be irradiated with radiation, which may damage not only the tissue but also the surrounding tissue. Radiation can change the nature of the tissue, often making it harder and less compressible. When a surgeon needs to staple across this tissue, the change in tissue hardness should be considered when selecting the staple reload type (for example, harder tissue requires staples formed higher).

[0306] When the lungs expand and contract, in areas with hard tissue, the lungs are not very flexible, so the amount of deformation is different compared to healthy tissue. Due to the tissue deformation force of this process 4850, these differences can be detected, and surgeons can be guided to place the stapler and select the color of the cartridge / reload.

[0307] In another aspect, a memory such as memory 134 can be configured to store the lung surface topography for known pressures and volumes. In this example, an imaging device such as imaging device 4872 can emit a light pattern to determine the topography of the patient's lung surface at a known first pressure or volume. A surgical system such as surgical system 2100 can be configured to compare a first determined topography at a known first pressure or volume with the topography stored in memory 134 for a given first pressure or volume. Based on this comparison, the visualization system can be configured to indicate the likelihood of tissue abnormalities in only a single state. The visualization system can focus on these regions of abnormality likelihood and proceed to determine the topography of the patient's lung surface at a second known pressure or volume. The visualization system can compare the second determined surface topography with the topography stored in memory for a given second pressure or volume, as well as the topography determined at the known first pressure or volume. If the visualization system determines an area of likelihood of abnormality that overlaps with the first determined area of likelihood of abnormality, the visualization system can be configured to indicate the overlapping area as a likelihood of abnormality with higher confidence based on the comparison at the known first and second pressures or volumes.

[0308] In addition to the above, the PO from the ventilator 2 measurement values can compare the inflated lung volume such as V 2 and the deflated lung volume such as V 1 . The volume comparison uses the EKG data to compare inspiration and expiration such that it can be compared with the blood oxygen concentration. Also, this can be compared with the anesthesia gas exchange measurement values to determine the respiratory volume versus oxygen uptake versus sedation level. In addition, the EKG data provides approximate frequency data regarding arterial deformation. This frequency data, within a similar frequency range in which the surface geometry changes, can help identify important vascular structures.

[0309] In another embodiment, current tracking / treatment information can be compared to preoperative planning simulations. In the case of a difficult surgery or a high-risk procedure, the clinician can utilize a preoperative patient scan to simulate the surgical approach. This dataset can be compared to real-time measurements on a display such as display 146 and can help enable the surgeon to follow a particular preoperative plan based on training runs. This requires a function to match reference landmarks between the preoperative scan / simulation and the current visualization. In one way, it may be sufficient to use boundary tracking of the object. Insights regarding how the current device-tissue interaction compares to previous interactions (per patient), or predicted interactions (database, or past patients), for tissue type discrimination, relative tissue deformation assessment, or subsurface structural differences, can be stored in a memory such as memory 134.

[0310] In one embodiment, the surface geometry can be responsive to the instrument position. If the change in surface geometry is not measured for each change in instrument position, a surface reference can be selected. When the instrument interacts with the tissue and deforms the surface geometry, the change in surface geometry responsive to the instrument position can be calculated by the surgical system. For a given change in instrument position upon contact with the tissue, the change in tissue geometry may be different in regions that include subsurface structures such as important structure 4884 and regions that do not include such structures such as subsurface void 4886. In an example such as a chest surgery, this may be in the parenchymal tissue only, whereas it may also be above the airway. The running average of the change in instrument position and the change in surface geometry can be calculated by the surgical system for a given patient, tailored to the particular patient-specific differences, using the surgical visualization system, or the value can be compared to a second set of previously collected data.

[0311] Exemplary clinical uses The various surgical visualization systems disclosed herein can be employed in one or more of the following clinical applications. The following clinical applications are not exhaustive and are merely exemplary uses for one or more of the various surgical visualization systems disclosed herein.

[0312] As disclosed herein, surgical visualization systems can be employed in many different types of procedures in different specialties such as, for example, urology, gynecology, oncology, colorectal, thoracic, bariatric / gastric, and hepatopancreatobiliary (HPB). For example, in urological surgeries such as prostatectomy, the urinary tract may be detected within fat or connective tissue and / or nerves may be detected, for example, within fat. For example, in gynecological oncology surgeries such as hysterectomy and colorectal surgeries such as low anterior resection (LAR), the ureters may be detected, for example, within fat and / or connective tissue. For example, in thoracic surgeries such as lobectomy, blood vessels may be detected within the lung or connective tissue and / or nerves may be detected within connective tissue (e.g., esophageal fistula formation). In bariatric surgery, blood vessels may be detected within fat. For example, in HPB surgeries such as hepatectomy or pancreatomy, blood vessels may be detected within fat (extrahepatic), connective tissue (extrahepatic), and bile ducts may be detected within parenchymal tissue (liver or pancreas).

[0313] In one example, a clinician may wish to remove a uterine leiomyoma. From a preoperative magnetic resonance imaging (MRI) scan, the clinician can know that the uterine leiomyoma is located on the surface of the intestine. Thus, the clinician may wish to know during the surgery which tissue constitutes a part of the intestine and which tissue constitutes a part of the rectum. In such an example, the surgical visualization system can, as disclosed herein, indicate different types of tissue (intestine versus rectum) and communicate that information to the clinician via an imaging system. Further, the imaging system can determine and communicate the proximity of a surgical device to the selected tissue. In such an example, the surgical visualization system can improve treatment efficiency without significant complications.

[0314] In another example, a clinician (e.g., a gynecologist) may remain away from a particular anatomical region to avoid getting too close to important structures, and thus the clinician may not remove all of, for example, endometriosis. A surgical visualization system, as disclosed herein, can enable a gynecologist to reduce the risk of getting too close to important structures so that the surgical device can get close enough to remove all of the endometriosis and improve the patient's outcome (democratized surgery). Such a system can enable a surgeon to "keep moving" during surgery rather than repeatedly stopping and restarting to identify areas to avoid, for example, during the application of therapeutic energy such as ultrasonic or electro-surgical energy in particular. In gynecological applications, the uterine arteries and ureters are important structures, and the system can be particularly useful for hysterectomy and endometrial surgery, taking into account the presentation and / or thickness of the tissues involved.

[0315] In another example, a clinician may risk incising a blood vessel at a location that may affect the blood supply to leaves other than the targeted leaf due to being too close. Further, anatomical differences between patients may result in incising blood vessels (e.g., branched blood vessels) that affect different leaves based on a particular patient. A surgical visualization system, as disclosed herein, can enable the correct identification of the blood vessel at the desired location, thereby enabling the clinician to reliably incise the appropriate anatomical object. For example, the system can confirm that the correct blood vessel is in the correct position, after which the clinician can safely divide the blood vessel.

[0316] In another example, due to the uncertain anatomical structure of blood vessels, a clinician may make multiple incisions before making an incision at the best location. However, since more incisions may increase the risk of bleeding, it is desirable to make an incision at the best location in the first step. As disclosed herein, a surgical visualization system can minimize the number of incisions by indicating the correct blood vessels and the best location for the incision. For example, the ureter and the ligament are dense and present unique challenges during an incision. In such an example, it may be particularly desirable to minimize the number of incisions.

[0317] In another example, a clinician (e.g., an oncological surgeon) who removes cancerous tissue may wish to know the identification of important structures, the localization of cancer, the diagnosis of the cancer stage, and / or the evaluation of the normality of the tissue. Such information goes beyond what a clinician can see "with the naked eye". As disclosed herein, a surgical visualization system can determine such information and / or communicate such information to the clinician during surgery to enhance intraoperative decision-making and improve surgical outcomes. In certain examples, the surgical visualization system can be compatible with minimally invasive surgery (MIS), laparotomy, and / or a robotic approach using either an endoscope or an exoscope, for example.

[0318] In another example, a clinician (e.g., an oncological surgeon) may wish to turn off one or more warnings regarding the approach of a surgical instrument to one or more important structures in order to avoid being overly cautious during surgery. In other examples, a clinician may wish to receive a specific type of warning, such as tactile feedback (e.g., vibration / buzzer) indicating proximity and / or "no-fly zone", so as to remain sufficiently far away from one or more important structures. As disclosed herein, a surgical visualization system can provide adaptability, for example, based on the clinician's experience and / or the desired aggressiveness of the procedure. In such an example, the system provides a balance between "knowing too much" and "knowing enough" to predict and avoid important structures. The surgical visualization system can assist in planning the next step(s) during surgery.

[0319] Various aspects of the subject matter described in this specification ...

Claims

1. 1. A surgical system for use in a surgical procedure, the surgical system comprising: A surgical visualization system; A control circuit comprising: estimating at least one of a surface area and a volume of the lung without any portion resected based on visualization data from the surgical visualization system, and estimating a first maximum lung capacity of the lung without any portion resected based on the estimated at least one of the surface area and the volume of the lung; estimating a second maximum lung volume of the lung after a portion of the lung is resected by a resection path determined based on the visualization data from the surgical visualization system; and proposing the ablation path if a ratio of the second maximum lung capacity to the first maximum lung capacity is equal to or greater than a predetermined value; and a control circuit configured to:

1. A surgical system comprising:

2. 1. A surgical system for use in a surgical procedure, the surgical system comprising: A surgical visualization system; a ventilator for measuring a first actual maximum lung volume of the lung without any part of the lung being resected; A control circuit comprising: estimating a second maximum lung volume of the lung after a portion of the lung is resected by a resection path determined based on visualization data from the surgical visualization system; and proposing the ablation path when a ratio of the second maximum lung capacity to the first maximum lung capacity measured by the ventilator is equal to or greater than a predetermined value; and a control circuit configured to:

1. A surgical system comprising:

3. The actual maximum lung capacity or PCO of the lung without any part being resected ２ and measuring the actual maximum lung capacity or PCO of the lung after the portion of the lung is resected by the resection path. ２ 13. The surgical system of claim 1, further comprising a ventilator that measures:

4. The surgical system of claim 2 , wherein the ventilator measures a real maximum lung capacity of the lung after the portion of the lung is resected by the resection pathway.

5. 3. The surgical system of claim 1, wherein the estimation of the second maximum lung capacity of the lung is performed by estimating at least one of a surface area and a volume of the lung after the portion of the lung is resected based on the visualization data from the surgical visualization system, and estimating the second maximum lung capacity of the lung after the portion of the lung is resected based on the estimated at least one of the surface area and the volume of the lung after the portion of the lung is resected.

6. The surgical system of claim 1 or 2, wherein the control circuitry is further configured to detect air leaks in the lung after the portion of the lung is resected.

7. The surgical system of claim 6 , further comprising: a surgical visualization system configured to detect said air leak using dynamic visualization data of said lung from said surgical visualization system after said portion of said lung is resected.

Images