Surgical system control based on multiple sensed parameters

Abstract

Various surgical control systems are disclosed for controlling a surgical instrument. The surgical instrument can be controlled according to sensor data. However, some sensory data may indicate different states or conditions of tissue acted upon by the surgical instrument and, therefore, may be inconclusive without additional data. Thus, the surgical control system can utilize multiple independent sensor data measurements to cooperatively control the surgical instrument or another coupling device. The present disclosure relates to a control system for a surgical instrument, the control system including: an imaging system including an emitter configured to emit electromagnetic radiation and an image sensor configured to receive reflected electromagnetic radiation indicative of a first tissue parameter; a sensor configured to detect a second tissue parameter of the tissue acted upon by the surgical instrument; and control circuitry coupled to the imaging system and the sensor, the control circuitry configured to determine a state of the tissue based on a combination of the first tissue parameter and the second tissue parameter, and to control the surgical instrument according to the determined state of the tissue.

Description

Background Art

[0001] Surgical systems often incorporate an imaging system that enables a clinician to observe a surgical site and / or one or more portions thereof on one or more displays such as a monitor. The display may be local and / or remote with respect to the operating room. The imaging system can include a scope with a camera that observes the surgical site and transmits an image to a display viewable by the clinician. Scopes include, but are not limited to, arthroscopes, endovascular scopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-ureteroscopes, sigmoidoscopes, thoracoscopes, ureteroscopes, and external scopes. The imaging system can be limited by information that can recognize and / or communicate to the clinician. For example, certain hidden structures, physical contours, and / or dimensions within a three-dimensional space may not be recognizable during surgery with a particular imaging system. In addition, certain imaging systems may not be able to communicate and / or transmit certain information to the clinician during surgery.

Summary of the Invention

Means for Solving the Problems

[0002] In one general aspect, a control system for a surgical instrument is disclosed. The control system includes an imaging system, a sensor configured to detect a second tissue parameter of tissue acted upon by the surgical instrument, and a control circuit coupled to the imaging system and the sensor. The imaging system includes an emitter configured to emit electromagnetic radiation and an image sensor configured to receive the reflected electromagnetic radiation indicative of the first tissue parameter. The control circuit is configured to determine a state of the tissue based on a combination of the first tissue parameter and the second tissue parameter and to control the surgical instrument according to the determined state of the tissue.

[0003] In another general aspect, a control system for a surgical instrument is disclosed. The surgical instrument includes a sensor configured to detect a second parameter associated with the surgical instrument. The control system includes an imaging system and a control circuit coupled to the imaging system and configurable to couple to the sensor. The imaging system includes an emitter configured to emit electromagnetic radiation and an image sensor configured to receive the reflected electromagnetic radiation indicative of a first parameter of the tissue. The control circuit is configured to determine a state of the tissue based on a combination of the first parameter and the second parameter and to control the surgical instrument according to the determined state of the tissue.

[0004] In yet another general aspect, an imaging system, a surgical device, and a surgical hub configurable to couple to a surgical instrument are disclosed. The imaging system includes an emitter configured to emit electromagnetic radiation at a surgical site and an image sensor configured to receive the reflected electromagnetic radiation from the surgical site, the reflected electromagnetic radiation indicative of a first parameter associated with the tissue, and the surgical device includes a sensor configured to detect a second parameter associated with the surgical device. The surgical hub includes a control circuit configured to receive a first measurement of the first parameter from the imaging system, to receive a second measurement of the second parameter from the sensor, to determine a state of the tissue based on a combination of the first parameter and the second parameter, and to control the surgical instrument according to the determined state of the tissue. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The novel features of the various aspects are set forth with particularity in the appended "Claims." However, the described aspects, both as to their construction and the manner of operation, together with further objects and advantages thereof, may best be understood by reference to the following description taken in conjunction with the accompanying drawings.

Figure 1

Figure 2

Figure 2A

Figure 2B

Figure 2C

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 13C

Figure 13D

Figure 13E

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33A

Figure 33B

Figure 34A

Figure 34B

Figure 34C

Figure 35

Figure 36

Figure 37

Figure 38

DETAILED DESCRIPTION OF THE INVENTION

[0006] The applicant of the present application owns the following U.S. patent applications filed simultaneously, the entire contents of each of which are incorporated herein by reference. · Attorney Docket No. END9228USNP1 / 190580 - 1M, Title of Invention "METHOD OF USING IMAGING DEVICES IN SURGERY", · Attorney Docket No. END9227USNP1 / 190579 - 1, Title of Invention "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM", · Attorney Docket No. END9225USNP1 / 190577-1, Invention Title "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS", · Attorney Docket No. END9224USNP1 / 190576-1, Invention Title "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS", · Attorney Docket No. END9223USNP1 / 190575-1, Invention Title "SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA", · Attorney Docket No. END9222USNP1 / 190574-1, Invention Title SURGICAL SYSTEMS FOR GENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED, · Attorney Docket No. END9221USNP1 / 190573-1, Invention Title "SURGICAL SYSTEM FOR OVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE DIMENSIONAL CONSTRUCT OF AN ORGAN", · Attorney Docket No. END9220USNP1 / 190572-1, Invention Title "SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS", · Attorney Docket No. END9219USNP1 / 190571-1, Invention Title "SYSTEM AND METHOD FOR DETERMINING,ADJUSTING,AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE", · Attorney Docket No. END9218USNP1 / 190570-1, Invention Title "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT", · Attorney Docket No. END9217USNP1 / 190569-1, Invention Title "DYNAMIC SURGICAL VISUALIZATION SYSTEMS", and · Attorney Docket No. END9216USNP1 / 190568-1, Invention Title "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM".

[0007] The applicant of this application owns the following U.S. patent applications filed on March 15, 2019, the entire contents of each of which are incorporated herein by reference. · U.S. Patent Application No. 16 / 354,417, Invention Title "INPUT CONTROLS FOR ROBOTIC SURGERY", · U.S. Patent Application No. 16 / 354,420, Invention Title "DUAL MODE CONTROLS FOR ROBOTIC SURGERY", · U.S. Patent Application No. 16 / 354,422, Invention Title "MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY", · U.S. Patent Application No. 16 / 354,440, Invention Title "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY", · U.S. Patent Application No. 16 / 354,444, Invention Title "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE", · U.S. Patent Application No. 16 / 354,454, titled "ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS", · U.S. Patent Application No. 16 / 354,461, titled "SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS", · U.S. Patent Application No. 16 / 354,470, titled "SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS", · U.S. Patent Application No. 16 / 354,474, titled "ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES", · U.S. Patent Application No. 16 / 354,478, titled "ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK", and · U.S. Patent Application No. 16 / 354,481, titled "JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS".

[0008] The applicant of the present application also owns the following U.S. patent applications filed on September 11, 2018, and the entire contents of each of these are incorporated herein by reference. · U.S. Patent Application No. 16 / 128,179, titled "SURGICAL VISUALIZATION PLATFORM", · U.S. Patent Application No. 16 / 128,180, titled "CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE", · U.S. Patent Application No. 16 / 128,198, titled "SINGULAR EMR SOURCE EMITTER ASSEMBLY", · U.S. Patent Application No. 16 / 128,207, titled "COMBINATION EMITTER AND CAMERA ASSEMBLY", · U.S. Patent Application No. 16 / 128,176, titled "SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES", · U.S. Patent Application No. 16 / 128,187, titled "SURGICAL VISUALIZATION OF MULTIPLE TARGETS", · U.S. Patent Application No. 16 / 128,192, titled "VISUALIZATION OF SURGICAL DEVICES", · U.S. Patent Application No. 16 / 128,163, titled "OPERATIVE COMMUNICATION OF LIGHT", · U.S. Patent Application No. 16 / 128,197, titled "ROBOTIC LIGHT PROJECTION TOOLS", · U.S. Patent Application No. 16 / 128,164, titled "SURGICAL VISUALIZATION FEEDBACK SYSTEM", · U.S. Patent Application No. 16 / 128,193, titled "SURGICAL VISUALIZATION AND MONITORING", · U.S. Patent Application No. 16 / 128,195, titled "INTEGRATION OF IMAGING DATA", · U.S. Patent Application No. 16 / 128,170, titled "ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS", · U.S. Patent Application No. 16 / 128,183, titled "SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS", · U.S. Patent Application No. 16 / 128,172, titled "ROBOTIC SYSTEM WITH SEPARATE PHOTOACOUSTIC RECEIVER", and · U.S. Patent Application No. 16 / 128,185, titled "FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION".

[0009] The applicant of the present application also owns the following U.S. patent applications filed on March 29, 2018, and the entire contents of each of them are incorporated herein by reference. · U.S. Patent Application No. 15 / 940,627, titled "DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", currently U.S. Patent Application Publication No. 2019 / 0201111, · U.S. Patent Application No. 15 / 940,676, titled "AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", currently U.S. Patent Application Publication No. 2019 / 0201142, · U.S. Patent Application No. 15 / 940,711, titled "SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS", currently U.S. Patent Application Publication No. 2019 / 0201120, and · U.S. Patent Application No. 15 / 940,722, titled "CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY", currently U.S. Patent Application Publication No. 2019 / 0200905.

[0010] The applicant of the present application owns the following U.S. patent applications filed on December 4, 2018, and each of these disclosures is incorporated herein by reference in its entirety. · U.S. Patent Application No. 16 / 209,395, titled "METHOD OF HUB COMMUNICATION", currently U.S. Patent Application Publication No. 2019 / 0201136, · U.S. Patent Application No. 16 / 209,403, titled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB", currently U.S. Patent Application Publication No. 2019 / 0206569, · U.S. Patent Application No. 16 / 209,407, titled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", currently U.S. Patent Application Publication No. 2019 / 0201137, · U.S. Patent Application No. 16 / 209,416, titled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS", currently U.S. Patent Application Publication No. 2019 / 0206562, · U.S. Patent Application No. 16 / 209,423, titled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", currently U.S. Patent Application Publication No. 2019 / 0200981, · U.S. Patent Application No. 16 / 209,427, titled "METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES", currently U.S. Patent Application Publication No. 2019 / 0208641, · U.S. Patent Application No. 16 / 209,433, titled "METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB", currently U.S. Patent Application Publication No. 2019 / 0201594, · U.S. Patent Application No. 16 / 209,447, titled "METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB", currently U.S. Patent Application Publication No. 2019 / 0201045, · U.S. Patent Application No. 16 / 209,453, titled "METHOD FOR CONTROLLING SMART ENERGY DEVICES", currently published as U.S. Patent Application Publication No. 2019 / 0201046, · U.S. Patent Application No. 16 / 209,458, titled "METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE", currently published as U.S. Patent Application Publication No. 2019 / 0201047, · U.S. Patent Application No. 16 / 209,465, titled "METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION", currently published as U.S. Patent Application Publication No. 2019 / 0206563, · U.S. Patent Application No. 16 / 209,478, titled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE", currently published as U.S. Patent Application Publication No. 2019 / 0104919, · U.S. Patent Application No. 16 / 209,490, titled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION", currently published as U.S. Patent Application Publication No. 2019 / 0206564, and · U.S. Patent Application No. 16 / 209,491, titled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS", currently published as U.S. Patent Application Publication No. 2019 / 0200998.

[0011] Before detailing the various aspects of the surgical visualization platform, it should be noted that the exemplary embodiments are not limited to the details of the structure and arrangement of the components illustrated in the accompanying drawings and specification in their application or use. The exemplary embodiments may be implemented in or incorporated into other aspects, variations, and modifications, and may be carried out or executed in various ways. Further, unless otherwise specified, the terms and expressions used herein are selected for the purpose of explaining the exemplary embodiments for the convenience of the reader and are not intended to limit them. Additionally, it should be understood that one or more of the aspects, expressions of aspects, and / or embodiments described below can be combined with any one or more of the other aspects, expressions of aspects, and / or embodiments described below.

[0012] Surgical visualization system The present disclosure is directed to a surgical visualization platform that utilizes "digital surgery" to obtain additional information regarding a patient's anatomical structure and / or surgical procedure. The surgical visualization platform is further configured to communicate data and / or information to one or more clinicians in a useful manner. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomical structure and / or surgical procedure.

[0013] "Digital surgery" can include robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analysis for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and the like. Although the various surgical visualization platforms described herein can be used in combination with robotic surgical systems, the surgical visualization platform is not limited to use with robotic surgical systems. In certain examples, high-level surgical visualization can be performed without a robot and / or in a state where robotic assistance is limited and / or optional. Similarly, digital surgery can be performed without a robot and / or in a state where robotic assistance is limited and / or optional.

[0014] In certain examples, a surgical system incorporating a surgical visualization platform may enable smart incisions to identify and avoid critical structures. Critical structures can include anatomical structures, such as arteries like the ureter, superior mesenteric artery, veins like the portal vein, nerves like the phrenic nerve, and / or tumors, among other anatomical structures. In other examples, critical structures can be foreign structures in the anatomical field, such as surgical devices, surgical fasteners, clips, clasps, sponges, bands, and / or plates. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Exemplary critical structures are further described herein. Smart incision techniques can provide improved intraoperative guidance for incisions and / or enable smarter decisions, for example, through important anatomical detection and avoidance techniques.

[0015] A surgical system incorporating a surgical visualization platform may also enable smart anastomosis, which provides more reliable anastomosis at an optimal location through an improved workflow. Cancer localization methods may also be improved by the various surgical visualization platforms and procedures described herein. For example, cancer localization methods can identify and track the location, orientation, and margins of cancer. In certain examples, cancer localization methods can correct for the movement of instruments, the patient, and / or the patient's anatomical structures during surgery and re-direct the clinician to the target point.

[0016] In certain aspects of the present disclosure, a surgical visualization platform may provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize the type and health of tissue without the need for physical tactile sensation, particularly during incision and / or placement of a stapling device within the tissue. Certain tissue characterization techniques described herein can be utilized without using ionizing radiation and / or contrast agents. With regard to lymph node diagnosis and mapping, a surgical visualization platform can, for example, preoperatively locate, map, and ideally diagnose lymphatic systems and / or lymph nodes involved in cancer diagnosis and staging.

[0017] During surgery, the information available to the clinician via "naked eye" and / or imaging systems may provide an incomplete view of the surgical site. For example, certain structures such as those embedded in or covered by an organ may be at least partially hidden from view, i.e., may not be visible. In addition, certain dimensions and / or relative distances may be difficult to confirm with existing sensor systems and / or may be difficult to grasp "naked eye". Furthermore, certain structures may move preoperatively (e.g., prior to surgery but after preoperative scans) and / or during surgery. In such cases, the clinician may not be able to accurately determine the location of important structures during surgery.

[0018] When the location of important structures is uncertain and / or the proximity between important structures and surgical tools is unclear, the clinician's decision-making process can be inhibited. For example, the clinician can avoid a particular area to avoid inadvertent dissection of important structures, but the avoided area may be unnecessarily large and / or at least partially mislocated. Due to uncertainty and / or excessive / cautious concern, the clinician may not be able to reach a particular desired area. For example, even if an important structure is not in that particular area and / or the clinician's actions in that particular area have no adverse effects, due to excessive caution, the clinician may try to avoid the important structure and leave behind a portion of the tumor and / or other unwanted tissue. In certain cases, surgical outcomes may improve as knowledge and / or certainty increase, which may enable the surgeon to be more accurate and, in certain cases, more conservative / more aggressive with respect to a particular anatomical area.

[0019] In various aspects, the present disclosure provides a surgical visualization system for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision-making and improved surgical outcomes. In various aspects, the disclosed surgical visualization system provides advanced visualization capabilities beyond what a clinician can see “with the naked eye” and / or what an imaging system can recognize and / or communicate to the clinician. The various surgical visualization systems can improve outcomes in various instances by reinforcing and enhancing what a clinician can know prior to tissue manipulation (e.g., incision).

[0020] For example, a visualization system can include a first light emitter configured to emit a plurality of spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers, or sensors, configured to detect visible light, molecular responses to spectral waves (spectroscopic imaging), and / or the light pattern. Throughout the following disclosure, note that any reference to "light" can include EMR or photons in the visible portion and / or non-visible portion of the electromagnetic radiation (EMR) wavelength spectrum, unless specifically referred to otherwise with respect to visible light. A surgical visualization system can also include an imaging system and a control circuit that signal communicates between the receiver and the imaging system. Based on the output from the receiver, the control circuit can determine a geometric surface map of the visible surface at the surgical site, i.e., a three-dimensional surface topography, and one or more distances to the surgical site. In certain examples, the control circuit can determine one or more distances to at least partially hidden structures. Additionally, the imaging system can communicate the geometric surface map and the one or more distances to the clinician. In such an example, an extended view of the surgical site provided to the clinician can provide a view of structures hidden within the background associated with the surgical site. For example, the imaging system can virtually extend hidden structures onto the geometric surface map of the obscuring and / or obstructing tissue, similar to lines drawn on the ground to indicate subsurface utility piping. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to the visible obstructive tissue and / or to at least partially hidden structures, and / or the depth of hidden structures beneath the visible surface of the obstructive tissue. For example, the visualization system can determine the distance to an extension line on the surface of the visible tissue and communicate that distance to the imaging system.

[0021] In various aspects of the present disclosure, a surgical visualization system for intraoperative identification and avoidance of critical structures is disclosed. Such a surgical visualization system can provide beneficial information to a clinician during a surgical procedure. As a result, while knowing that the surgical visualization system is tracking critical structures, such as, for example, the ureter, certain nerves, and / or important blood vessels, that may be approached during an incision, the clinician can maintain control throughout the surgical procedure. In one aspect, the surgical visualization system can provide an indication to the clinician for a sufficient amount of time for the clinician to pause and / or slow down the surgical technique and evaluate the proximity to the structure to prevent inadvertent damage to the critical structure. The surgical visualization system provides an ideal, optimized, and / or customizable amount of information to the clinician to avoid inadvertent damage to healthy tissue and / or critical structures while allowing the clinician to manipulate the tissue as a whole surely and / or quickly, thereby minimizing the risk of damage caused by the surgical procedure.

[0022] FIG. 1 is a schematic diagram of a surgical visualization system 100 according to at least one aspect of the present disclosure. The surgical visualization system 100 can create a visual representation of critical structures 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In a particular example, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near real-time information regarding proximity data, dimensions, and / or distances during a surgical procedure. The surgical visualization system 100 is configured to facilitate intraoperative identification of critical structures and / or avoidance of critical structures 101 by a surgical device. For example, by identifying critical structures 101, the clinician can avoid operating a surgical device around critical structures 101 and / or a predetermined proximal region of critical structures 101 during a surgical procedure. The clinician can avoid, for example, incisions in or near critical structures 101, such as, for example, veins, arteries, nerves, and / or blood vessels that are identified as critical structures 101. In various examples, critical structures 101 may be determined on a patient-by-patient and / or procedure-by-procedure basis.

[0023] The surgical visualization system 100 incorporates tissue identification and geometric surface mapping in combination with a distance sensor system 104. The combined surgical visualization system 100 can determine the location of important structures 101 within an anatomical field and / or the proximity of a surgical device 102 to the surface 105 of visible tissue and / or important structures 101. Additionally, the surgical visualization system 100 includes an imaging system that includes an imaging device 120, such as a camera, configured to provide, for example, a real-time image of the surgical site. In various examples, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectral camera) configured to detect reflected spectral waveforms and generate a spectral cube of an image based on the molecular response to different wavelengths. Images from the imaging device 120 can be provided to a clinician, and in various aspects of the present disclosure, additional information can be added based on tissue identification, situation mapping, and the distance sensor system 104. In such an example, the surgical visualization system 100 includes a plurality of subsystems, namely, an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems can cooperate to provide the clinician with advanced data synthesis and integrated information during the surgery.

[0024] The imaging device can include, for example, a camera or an imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In various aspects of the present disclosure, the imaging system can include an imaging device, such as an endoscope. Additionally or alternatively, the imaging system can include an imaging device, such as an arthroscope, a vascular endoscope, a bronchoscope, a choledochoscope, a colonoscope, a cystoscope, a duodenoscope, an enteroscope, an esophagogastroduodenoscope (gastroscope), a laryngoscope, a nasopharyngo-neproscope, a sigmoidoscope, a thoracoscope, a ureteroscope, or an externaloscope. In other examples, such as for open surgery applications, the imaging system may not include a scope.

[0025] In various aspects of the present disclosure, the tissue identification subsystem can be achieved with a spectral imaging system. The spectral imaging system can rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, entitled "SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING," issued on March 1, 2016, which is hereby incorporated by reference in its entirety.

[0026] In various aspects of the present disclosure, the surface mapping subsystem can be achieved with an optical pattern system, as further described herein. The use of optical patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be utilized in the surgical visualization systems described herein.

[0027] Structured light is the process of projecting a known pattern (often a grid or horizontal bar) onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled "DEPICTION SYSTEM," published on September 7, 2017, disclose surgical systems that include a light source and a projector for projecting an optical pattern. U.S. Patent Application Publication No. 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled "DEPICTION SYSTEM," published on September 7, 2017, are hereby incorporated by reference in their entireties.

[0028] In various aspects of the present disclosure, the distance determination system can be incorporated into a surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of a visible surface and determine various distances to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to identified tissue (or other structures) at the surgical site.

[0029] FIG. 2 is a schematic diagram of a control system 133 that can be utilized with a surgical visualization system 100. The control system 133 includes a control circuit 132 that communicates with a memory 134 in signal. The memory 134 stores instructions executable by the control circuit 132 to determine and / or recognize critical structures (e.g., the critical structure 101 of FIG. 1), determine and / or calculate one or more distances and / or three-dimensional digital displays, and communicate specific information to one or more clinicians. For example, the memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of the logics 136, 138, 140, and 141. The control system 133 also includes an imaging system 142 having one or more cameras 144 (such as the imaging device 120 of FIG. 1), one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 can include one or more image sensors 135 (e.g., among others, visible light, spectral imaging device, three-dimensional lens) for receiving signals from various light sources that emit light in various visible and invisible spectra. The display 146 can include one or more screens or monitors for depicting real, virtual, and / or virtually extended images and / or information to one or more clinicians.

[0030] In various aspects, the central part of camera 144 is image sensor 135. Generally, state-of-the-art image sensors 135 are solid-state electronic devices that contain up to several million discrete light-detecting sites called pixels. Image sensor 135 technology is classified into one of two categories: charge-coupled device (CCD) and complementary metal-oxide-semiconductor (CMOS) imaging devices. More recently, short-wave infrared (SWIR) has been a new technology in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCOMS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 have sensitivity to wavelengths in the range of approximately 350 - 1050 nm, although this range is typically 400 - 1000 nm. CMOS sensors generally have higher sensitivity to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish colors. Thus, there are two types of color CCD cameras: one-chip and three-chip. One-chip color CCD cameras provide a common low-cost imaging solution, using a mosaic (e.g., Bayer) optical filter to separate the input light into a series of colors and an interpolation algorithm to resolve a full-color image. Each color is then directed to a different set of pixels. Three-chip color CCD cameras provide higher resolution by using a prism to direct each section of the incident spectrum to a different chip. Since each point in the object's space has separate RGB intensity values rather than using an algorithm to determine color, more accurate color reproduction is possible. Three-chip cameras provide very high resolution.

[0031] The control system 133 also includes a spectral light source 150 and a structured light source 152. In certain examples, a single light source can pulse-radiate the wavelengths of light within the range of the spectral light source 150 and the wavelengths of light within the range of the structured light source 152. Alternatively, a single light source can pulse-feed the wavelengths of light within the invisible spectrum (e.g., infrared spectrum light) and the wavelengths of light on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectral light source. In various examples, the tissue discrimination logic 140 can identify important structures via data from the spectral light source 150 received by the image sensor 135 portion of the camera 144. The surface mapping logic 136 can determine the contour of the surface of the visible tissue based on the reflected structured light. By time-of-flight measurement, the distance determination logic 141 can determine one or more distances to the visible tissue and / or the important structure 101. One or more outputs from the surface mapping logic 136, the tissue discrimination logic 140, and the distance determination logic 141 can be provided to the imaging logic 138 and combined, integrated, and / or overlaid so as to be communicated to the clinician via the display 146 of the imaging system 142.

[0032] Here, FIGS. 2A to 2C will be briefly described, and various aspects of the control circuit 132 for controlling various aspects of the surgical visualization system 100 will be described. Looking at FIG. 2A, a control circuit 400 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure is shown. The control circuit 400 can be configured to implement various processes described herein. The control circuit 400 can include a microcontroller including one or more processors 402 (e.g., microprocessors, microcontrollers) connected to at least one memory circuit 404. The memory circuit 404 stores machine-executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions for implementing various processes described herein. The processor 402 can be any one of a number of single-core or multi-core processors known in the art. The memory circuit 404 can include volatile and non-volatile storage media. The processor 402 may include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from the memory circuit 404 of the present disclosure.

[0033] FIG. 2B shows a combinational logic circuit 410 configured to control an aspect of the surgical visualization system 100 according to at least one aspect of the present disclosure. The combinational logic circuit 410 can be configured to implement various processes described herein. The combinational logic circuit 410 may include a finite state machine including combinational logic 412 configured to receive data associated with a surgical instrument or tool at input 414, process the data by combinational logic 412, and provide an output 416.

[0034] FIG. 2C shows a sequential logic circuit 420 configured to control aspects of a surgical visualization system 100 according to at least one aspect of the present disclosure. The sequential logic circuit 420 or combinatorial logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, combinatorial logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store the current state of the finite state machine. In certain examples, the sequential logic circuit 420 may be synchronous or asynchronous. The combinatorial logic 422 is configured to receive data associated with a surgical device or system from an input 426, process the data by the combinatorial logic 422, and provide an output 428. In other aspects, the circuit may include a combination of a processor (e.g., processor 402 of FIG. 2A) and a finite state machine that implements various processes herein. In other aspects, the finite state machine can include a combination of a combinatorial logic circuit (e.g., combinatorial logic circuit 410 of FIG. 2B) and the sequential logic circuit 420.

[0035] Referring again to the surgical visualization system 100 of FIG. 1, the critical structure 101 can be a target anatomical structure. For example, the critical structure 101 can be, among other anatomical structures, an artery such as the ureter, superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor. In other examples, the critical structure 101 can be a foreign object structure in the anatomical field, such as a surgical device, a surgical fastener, a clip, a retainer, a bovie, a band, and / or a plate. Exemplary critical structures are further described herein and in the aforementioned U.S. patent applications, such as U.S. Patent Application No. 16 / 128,192, entitled "VISUALIZATION OF SURGICAL DEVICES," filed on September 11, 2018, which is hereby incorporated by reference in its entirety.

[0036] In one aspect, the critical structure 101 can be embedded in the tissue 103. In other words, the critical structure 101 can be positioned beneath the surface 105 of the tissue 103. In such an example, the tissue 103 hides the critical structure 101 from the clinician's view. The critical structure 101 is also blocked from the view of the imaging device 120 by the tissue 103. The tissue 103 can be, for example, fat, connective tissue, adhesions, and / or an organ. In other examples, the critical structure 101 can be partially blocked from the field of view.

[0037] FIG. 1 also shows a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from the distal end of the shaft of the surgical device 102. The surgical device 102 can be any suitable surgical device, such as, for example, a cutting instrument, a stapler, a grasping instrument, a clip applier, and / or an energy device including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector. Additionally or alternatively, the surgical device 102 can include another imaging or diagnostic modality, such as, for example, an ultrasonic device. In one aspect of the present disclosure, the surgical visualization system 100 can be configured to achieve identification of one or more critical structures 101 and proximity of the surgical device 102 to the critical structure 101.

[0038] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include a plurality of lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 can be a hyperspectral, multispectral, or selective spectral camera, as further described herein. The imaging device 120 can also include a waveform sensor 122 (e.g., a spectral image sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 can include a right lens and a left lens that are used together to generate a three-dimensional image of the surgical site, render the three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site by simultaneously recording two two-dimensional images. Additionally or alternatively, the imaging device 120 can be configured to receive images showing the topography of visible tissue and the identification and location of hidden critical structures, as further described herein. For example, as shown in FIG. 1, the field of view of the imaging device 120 can be overlaid with a pattern of light (structured light) on the surface 105 of the tissue.

[0039] In one aspect, the surgical visualization system 100 may be incorporated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include rigid structural members 116 and joints 118 that can include servo motor control. The first robotic arm 112 is configured to operate the surgical device 102, and the second robotic arm 114 is configured to operate the imaging device 120. The robotic control unit can be configured to issue control movements to the robotic arms 112, 114 that can act on, for example, the surgical device 102 and the imaging device 120.

[0040] The surgical visualization system 100 also includes an emitter 106 configured to emit a pattern of light, such as stripes, grid lines, and / or dots, to enable determination of the topography or situation of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from an emitter 106 located, for example, on one of the surgical device 102 and / or the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is used to determine the surface 105 of the tissue 103 and / or the shape defined during the procedure by the movement of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and the various distances to the surface 105.

[0041] In one aspect, the imaging device 120 may also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 to reach the critical structure 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effect of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 can be configured to enable identification of the type of anatomical and / or physical structure, such as the critical structure 101. Identification of the critical structure 101 can be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be variable. The waveform sensor 122 and the optical waveform emitter 123 can include, for example, a multispectral imaging system and / or a selective spectral imaging system. In other examples, the waveform sensor 122 and the optical waveform emitter 123 can include, for example, a photoacoustic imaging system. In other examples, the optical waveform emitter 123 can be disposed on a separate surgical device from the imaging device 120.

[0042] Surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one aspect, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system including an emitter such as emitter 106 and a receiver 108, and may be disposed on the surgical device 102. In other examples, the time-of-flight emitter may be separate from the structured light emitter. In one general aspect, the emitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a coincidence sensor. The time-of-flight distance sensor system 104 can detect the "time of flight", that is, the time it takes for the laser light emitted by the emitter 106 to bounce back to the sensor portion of the receiver 108. By using a very narrow light source in the emitter 106, it is possible for the distance sensor system 104 to determine the distance to the surface 105 of the tissue 103 immediately in front of the distance sensor system 104. Referring further to FIG. 1, d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the surface 105 of the tissue. The distance sensor system 104 is used to determine the emitter-tissue distance d e . The device-tissue distance d t can be obtained from the known position of the emitter 106 on the shaft of the surgical device 102 relative to the distal end of the surgical device 102. In other words, if the distance between the emitter 106 and the distal end of the surgical device 102 is known, the device-tissue distance d t can be determined from the emitter-tissue distance d e . In a particular example, the shaft of the surgical device 102 can include one or more articulating joints and can be articulable with respect to the emitter 106 and the joystick. The articulating configuration can include, for example, a multi-articulating vertebra-like structure. In a particular example, a three-dimensional camera can be utilized to triangulate one or more distances to the surface 105.

[0043] In various examples, the receiver 108 for the time-of-flight distance sensor system 104 can be mounted on another surgical device instead of the surgical device 102. For example, the receiver 108 can be attached to a cannula or trocar that extends through the surgical device 102 to reach the surgical site. In still other examples, the receiver 108 for the time-of-flight distance sensor system 104 can be attached to another robotic control arm (e.g., robotic arm 114), a movable arm operated by another robot, and / or an operating room (OR) table or fixture. In a particular example, the imaging device 120 includes a time-of-flight receiver 108 that determines the distance from the emitter 106 to the surface 105 of the tissue 103 using the line between the emitter 106 on the surgical device 102 and the imaging device 120. For example, the distance d e can be triangulated based on the known positions of the emitter 106 (on the surgical device 102) and the receiver 108 (on the imaging device 120) of the time-of-flight distance sensor system 104. The three-dimensional position of the receiver 108 is known and / or can be aligned with respect to the intraoperative robotic coordinate plane.

[0044] In a particular example, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by the first robotic arm 112, and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by the second robotic arm 114. In other examples, the surgical visualization system 100 can be utilized separately from the robotic system. In such cases, the distance sensor system 104 may be independent of the robotic system.

[0045] In certain examples, one or more of the robotic arms 112, 114 may be separate from the main robotic system used in surgery. At least one of the robotic arms 112, 114 can be positioned and aligned in a particular coordinate system without servo motor control. For example, a closed-loop control system and / or multiple sensors for the robotic arm 110 can control and / or align the positions of the robotic arms 112, 114 relative to a particular coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 can be aligned relative to a particular coordinate system.

[0046] Referring further to FIG. 1, d w is the camera-important structure distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the important structure 101, and d A is the depth of the important structure 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the important structure 101). In various aspects, the time of flight of the optical waveform emitted from the optical waveform emitter 123 located on the imaging device 120 can be configured to determine the camera-important structure distance d w . The use of spectral imaging in combination with a time of flight sensor is further described herein. Further, referring now to FIG. 3, in various aspects of the present disclosure, the depth d of the important structure 101 relative to the surface 105 of the tissue 103 A is the distance d w , as well as the known positions of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120 (and thus the distance d x ) between them, and can be determined by triangulation, and the distance d e and d A , the sum of which is the distance d y can be determined.

[0047] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or range of waveforms) can be utilized to determine the camera-important structure distance dw can be determined, and the second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such an example, different waveforms can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0048] Additionally or alternatively, in certain examples, the distance d A can be determined from an ultrasound, a registered magnetic resonance imaging (MRI), or a computed tomography (CT) scan. In yet other examples, the distance d A can be determined by spectral imaging because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first way or by a first amount, and collagen can reduce the detection signal in a different second way or by a second amount.

[0049] Referring now to the surgical visualization system 160 of FIG. 4, the surgical device 162 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect the reflected waveform. The optical waveform emitter 123 can be configured to emit waveforms for determining the distances d t and d w from a common device such as the surgical device 162, as further described herein. In such an example, the distance d A from the surface 105 of the tissue 103 to the surface of the critical structure 101 can be determined as follows: d A = d w - d t .

[0050] As disclosed herein, various information regarding visible tissue, embedded critical structures, and surgical devices can be determined by utilizing a combination method incorporating one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays, in combination with an image sensor configured to detect spectral wavelengths and structured light arrays. Further, the image sensor can be configured to provide an image of the surgical site to the imaging system by receiving visible light. Logic or algorithms are used to discriminate information received from the time-of-flight sensors, spectral wavelengths, structured light, and visible light and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various examples, the imaging device 120 can include a plurality of image sensors.

[0051] Camera - Critical Structure Distance d w It can also be detected by one or more alternative methods. In one aspect, for example, fluorescence fluoroscopy visualization techniques such as indocyanine green (ICG) can be used to illuminate critical structures 201 as shown in FIGS. 6 - 8. The camera 220 can include two optical waveform sensors 222, 224 that simultaneously capture left and right images of the critical structure 201 (FIGS. 7A and 7B). In such an example, the camera 220 can depict the light of the critical structure 201 beneath the surface 205 of the tissue 203, and the distance d w can be determined by the known distance between sensors 222 and 224. In certain examples, the distance can be determined more accurately by utilizing two or more cameras or by moving the camera between multiple positions. In a particular aspect, one camera can be controlled by a first robotic arm and a second camera can be controlled by another robotic arm. In such a robotic system, one camera can be, for example, a passive camera on a passive arm. The passive arm and the camera thereon can be programmed, for example, to track another camera and maintain a specific distance and / or lens angle.

[0052] In yet another aspect, the surgical visualization system 100 may determine d using two separate waveform receivers (i.e., cameras / image sensors). w Referring now to FIG. 9, where an important structure 301 or its contents (e.g., a blood vessel or the contents of a blood vessel) may emit a signal 302, for example, by fluoroscopy, the actual position can be triangulated from two separate cameras 320a, 320b at known positions.

[0053] Referring now to FIGS. 10A and 10B, in another aspect, the surgical visualization system may use a dithering or moving camera 440 to determine the distance d wIt may be determined. The camera 440 is robotically controlled such that the three-dimensional coordinates at different positions of the camera 440 are known. In various examples, the camera 440 can pivot at the cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., the contents of a blood vessel or container) can emit signals, for example, by fluoroscopy, the actual position can be triangulated from the camera 440 that rapidly moves between two or more known positions. In FIG. 10A, the camera 440 moves axially along axis A. The camera 440 translates a distance d1 along axis A towards the critical structure 401 to a position shown as position 440' by, for example, moving back and forth on the robotic arm. When the camera 440 moves the distance d1 and the size of the image with respect to the critical structure 401 changes, the distance to the critical structure 401 can be calculated. For example, an axial translation of 4.28 mm (distance d1) may correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, the camera 440 can rotate or sweep along an arc between different positions. Referring now to FIG. 10B, the camera 440 moves axially along axis A and rotates by an angle θ3 about axis A. The pivot point 442 for the rotation of the camera 440 is disposed at the cannula / patient interface. In FIG. 10B, the camera 440 translates and rotates to position 440''. When the camera 440 moves and the edge of the image with respect to the critical structure 401 changes, the distance to the critical structure 401 can be calculated. In FIG. 10B, the distance d2 may be, for example, 9.01 mm and the angle θ3 may be, for example, 0.9 degrees.

[0054] FIG. 5 shows a surgical visualization system 500 that is similar in many respects to the surgical visualization system 100. In various examples, the surgical visualization system 500 may be a further exemplification of the surgical visualization system 100. Similar to the surgical visualization system 100, the surgical visualization system 500 includes a surgical device 502 and an imaging device 520. The imaging device 520 includes, for example, a spectral light emitter 523 configured to emit spectral light of a plurality of wavelengths to acquire a spectral image of a hidden structure. The imaging device 520 may also include, in various examples, electronic processing circuitry associated with a three-dimensional camera. The surgical visualization system 500 is shown being utilized during a surgical procedure to identify certain critical structures, such as ureter 501a and blood vessel 501b within an organ 503 (the uterus in this example) that is not visible from the surface and to facilitate avoiding them.

[0055] The surgical visualization system 500 is configured to determine an emitter-tissue distance d from an emitter 506 on the surgical device 502 to the surface 505 of the uterus 503 via structured light. e The surgical visualization system 500 is configured to extrapolate a device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 based on the emitter-tissue distance d. e The surgical visualization system 500 is also configured to determine a tissue-ureter distance d from the ureter 501a to the surface 505, and a camera-ureter distance d from the imaging device 520 to the ureter 501a. As described herein with respect to FIG. 1, for example, the surgical visualization system 500 can determine the distance d, for example, by spectral imaging and time-of-flight sensors. In various examples, the surgical visualization system 500 can determine (e.g., by triangulation) the tissue-ureter distance d (i.e., depth) based on other distances and / or surface mapping logic described herein. t The surgical visualization system 500 is also configured to determine a tissue-ureter distance d from the ureter 501a to the surface 505, and a camera-ureter distance d from the imaging device 520 to the ureter 501a. A The surgical visualization system 500 is also configured to determine a tissue-ureter distance d from the ureter 501a to the surface 505, and a camera-ureter distance d from the imaging device 520 to the ureter 501a. w As described herein with respect to FIG. 1, for example, the surgical visualization system 500 can determine the distance d, for example, by spectral imaging and time-of-flight sensors. w As described herein with respect to FIG. 1, for example, the surgical visualization system 500 can determine the distance d, for example, by spectral imaging and time-of-flight sensors. A (i.e., depth) based on other distances and / or surface mapping logic described herein.

[0056] Referring now to FIG. 11, there is shown a schematic diagram of a control system 600 for a surgical visualization system, such as surgical visualization system 100, for example. Control system 600 is a conversion system that integrates spectral signature tissue identification and structured light tissue positioning to identify these structures, particularly when important structures are obscured by other tissues, such as, for example, fat, connective tissue, blood, and / or other organs. Such techniques may also be useful for detecting tissue diversity, such as distinguishing tumors and / or diseased tissue from healthy tissue within an organ.

[0057] Control system 600 is configured to implement a hyperspectral imaging and visualization system in which molecular responses are used to detect and identify anatomical structures within the surgical field. Control system 600 includes conversion logic circuitry 648 for converting tissue data into information usable by a surgeon. For example, variable reflectance based on wavelengths for obscured substances can be utilized to identify important structures within anatomical structures. Further, control system 600 combines the identified spectral signature and structured light data within an image. For example, control system 600 can be used to create a three-dimensional data set for surgical use in a system having an enhanced image overlay. The technology can be used both intraoperatively and preoperatively using additional visual information. In various examples, control system 600 is configured to provide warnings to a clinician when in proximity to one or more important structures. Based on the surgical procedure and proximity to important structures, various algorithms can be employed to guide robotic automation and semi-automation approaches.

[0058] The shape and motion of the tissue are determined intraoperatively using a projected light array. Alternatively, a flash lidar may be utilized for surface mapping of the tissue.

[0059] The control system 600 is configured to detect critical structures, provide an image overlay of the critical structures, and measure the distance to the surface of the visible tissue and the distance to the embedded / covered critical structures. In other examples, the control system 600 can measure the distance to the surface of the visible tissue or detect critical structures and provide an image overlay of the critical structures.

[0060] The control system 600 includes a spectral control circuit 602. The spectral control circuit 602 can be, for example, a field programmable gate array (FPGA) or another suitable circuit configuration as described herein in connection with FIGS. 2A-2C. The spectral control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing and can utilize, for example, C / C++ code. The video input processor 606 receives video input terminals of control (metadata) data such as, for example, shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608 that includes, for example, hyperspectral video output terminals of interface control (metadata) data. The video output processor 608 provides the video output signal to an image overlay controller 610.

[0061] The video input processor 606 is connected to the patient-side camera 612 via the patient isolation circuit 614. As described above, the camera 612 includes a solid-state image sensor 634. The patient isolation circuit can include a plurality of transformers so that the patient is isolated from other circuits within the system. The camera 612 receives intraoperative images via the optical element 632 and the image sensor 634. The image sensor 634 can include, for example, a CMOS image sensor, or can include any of the image sensor technologies discussed herein in relation to FIG. 2. In one aspect, the camera 612 outputs an image with a 14-bit / pixel signal. It will also be understood that higher or lower pixel resolutions may be utilized without departing from the scope of the present disclosure. The separated camera output signal 613 is provided to a color RGB fusion circuit 616 that processes the camera output signal 613 using the hardware register 618 and the Nios2 coprocessor 620. A color RGB fusion output signal is provided to the video input processor 606 and the laser pulse control circuit 622.

[0062] The laser pulse control circuit 622 controls the laser light engine 624. The laser light engine 624 outputs light of a plurality of wavelengths (λ1, λ2, λ3... λ n ) including near-infrared (NIR). The laser light engine 624 can operate in a plurality of modes. In one aspect, the laser light engine 624 can operate in, for example, two modes. In the first mode, for example, the normal operation mode, the laser light engine 624 outputs an illumination signal. In the second mode, for example, the identification mode, the laser light engine 624 outputs RGBG light and NIR light. In various examples, the laser light engine 624 can operate in a polarization mode.

[0063] The light output 626 from the laser light engine 624 illuminates the target anatomical structure within the surgical site 627 during the operation. The laser pulse control circuit 622 also controls a laser pulse controller 628 for a laser pattern projector 630 that projects a laser light pattern 631, such as a grid or pattern of lines and / or dots, onto the surgical tissue or organ at the surgical site 627 at a predetermined wavelength (λ2). The camera 612 receives the patterned light and the reflected light output through the camera optical element 632. The image sensor 634 converts the received light into a digital signal.

[0064] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 and a video input module 636 for reading the laser light pattern 631 projected onto the target anatomical structure at the surgical site 627 by the laser pattern projector 630. The processing module 638 processes the laser light pattern 631 and outputs a first video output signal 640 representing the distance to the visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing the three-dimensional rendered shape of the tissue or organ of the target anatomical structure at the surgical site.

[0065] The first and second video output signals 640, 642 include data representing the positions of the important structures on the three-dimensional surface model provided to the integration module 643. In combination with the data from the video output processor 608 of the spectrum control circuit 602, the integration module 643 can determine the distance d A (Figure 1) (for example, via a triangulation algorithm 644), and the distance d A can be provided to the video overlay controller 610 via the video output processor 646. The aforementioned conversion logic can include a conversion logic circuit 648, an intermediate video monitor 652, and the camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0066] Using preoperative data 650 from a CT or MRI scan, specific three-dimensional deformable tissues can be aligned or registered in various examples. Such preoperative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610, such that such information can be overlaid on the images from the camera 612 and provided to the video monitor 652. The alignment of preoperative data is further described in this specification and in the aforementioned U.S. patent applications, such as U.S. Patent Application No. 16 / 128,195, titled "INTEGRATION OF IMAGING DATA," filed on September 11, 2018, which is hereby incorporated by reference in its entirety.

[0067] The video monitor 652 can output integrated / augmented images from the image overlay controller 610. The clinician can select and / or switch between different images on one or more monitors. On the first monitor 652a, the clinician can switch between (A) an image showing a three-dimensional rendering of the visible tissue and (B) an augmented image in which one or more hidden critical structures are drawn on top of the three-dimensional rendering of the visible tissue. On the second monitor 652b, the clinician can switch, for example, between distance measurements to one or more hidden critical structures and / or the surface of the visible tissue.

[0068] The control system 600 and / or its various control circuits can be incorporated into the various surgical visualization systems disclosed herein.

[0069] FIG. 12 shows a structured (or patterned) light system 700 according to at least one aspect of the present disclosure. As described herein, structured light, for example in the form of stripes or lines, is projected from a light source and / or projector 706 onto the surface 705 of a target anatomical structure to identify the shape and contour of the surface 705. For example, a camera 720, which may be similar in various respects to the imaging device 120 (FIG. 1), can be configured to detect the projected light pattern on the surface 705. By the way the projected pattern deforms when it hits the surface 705, it enables the vision system to calculate the depth and surface information of the target anatomical structure.

[0070] In certain examples, invisible (or imperceptible) structured light can be utilized, and the structured light is used without interfering with other computer vision tasks where the projected pattern may be disrupted. For example, infrared light or visible light at a very fast frame rate that repeats two exactly opposite patterns can be utilized to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0071] As described above, various different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that can be occluded so as not to be visualized by EMR in the visible portion of the spectrum, can be visualized using the various surgical visualization systems described herein. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissues based on various combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials within the tissue that are visualized based on the absorption coefficients of the tissue over various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the tissue that is visualized based on a particular combination of constituent materials. By way of illustration, FIG. 13A is a graph 2300 showing how the absorption coefficients of various biological materials vary over the EMR wavelength spectrum. In graph 2300, the vertical axis 2303 is the absorption coefficient of the biological material (e.g., cm-1 represents units, and the horizontal axis 2304 represents the EMR wavelength (e.g., in μm units). The graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials, and thus, the tissue types visualized by the surgical visualization system can be identified and distinguished according to the specific combination of detected constituent materials. Accordingly, the spectral imaging system can be configured to emit EMR at several different wavelengths, to determine the constituent materials of the tissue based on the absorption EMR absorption responses detected at different wavelengths, and then to characterize the tissue type based on the specific detected combination of constituent materials.

[0072] An example of the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures is shown in FIG. 13B. In FIG. 13B, a spectral emitter 2320 (e.g., spectral light source 150) is utilized by an imaging system to visualize a surgical site 2325. The EMR emitted by the spectral emitter 2320 and reflected from the tissue and / or structures of the surgical site 2325 is received by an image sensor 135 (FIG. 2) to enable visualization of the tissue and / or structures, which can be either visible (e.g., disposed on the surface of the surgical site 2325) or occluded (e.g., underlying other tissues and / or structures at the surgical site 2325). In this example, the imaging system 142 (FIG. 2) is based on spectral signatures characterized by different absorption characteristics (e.g., absorption coefficients) of the constituent materials for each of the different tissue / structure types, and can visualize a tumor 2332, an artery 2334, and various abnormalities 2338 (i.e., tissues that do not match known or expected spectral signatures). The visualized tissues and structures can be displayed on a display screen associated with or coupled to the imaging system 142, such as the imaging system display 146 (FIG. 2), the primary display 2119 (FIG. 18), the non-sterile display 2109 (FIG. 18), the hub display 2215 (FIG. 19), the device / instrument display 2237 (FIG. 19), etc.

[0073] Furthermore, the imaging system 142 can be configured to adjust or update the displayed surgical site visualization according to the identified tissue and / or structure type. For example, the imaging system 142 can display a margin 2330a associated with a tumor 2332 visualized on a display screen (e.g., display 146). The margin 2330a can indicate the area or amount of tissue to be resected to ensure complete removal of the tumor 2332. The control system 133 (FIG. 2) can be configured to control or update the dimensions of the margin 2330a based on the tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 has identified a plurality of abnormalities 2338 within the FOV. Accordingly, the control system 133 can adjust the displayed margin 2330a to a first updated margin 2330b having dimensions sufficient to encompass the abnormalities 2338. Furthermore, the imaging system 142 has also identified an artery 2334 that partially overlaps the initially displayed margin 2330a (as indicated by the highlighted region 2336 of the artery 2334). Accordingly, the control system 133 can adjust the displayed margin 2330a to a second updated margin 2330c having dimensions sufficient to encompass the relevant portion of the artery 2334.

[0074] Tissues and / or structures can also be imaged or characterized according to their reflection properties across the EMR wavelength spectrum, in addition to or instead of their absorption properties described above with respect to FIGS. 13A and 13B. For example, FIGS. 13C-13E show various graphs of the reflectance of different types of tissues or structures across different EMR wavelengths. FIG. 13C is a graphical representation 1050 of an exemplary ureter signature relative to a blocker. FIG. 13D is a graphical representation 1052 of an exemplary artery signature relative to a blocker. FIG. 13E is a graphical representation 1054 of an exemplary nerve signature relative to a blocker. The plots of FIGS. 13C-13E represent the reflectance as a function of wavelength (nm) of a particular structure (ureter, artery, and nerve) relative to the corresponding reflectance of fat, lung tissue, and blood at the corresponding wavelengths. These graphs are for illustrative purposes only, and it should be understood that other tissues and / or structures may have corresponding detectable reflectance signatures that enable the identification and visualization of the tissues and / or structures.

[0075] In various examples, the selected wavelengths for spectral imaging can be identified and utilized based on the critical structures and / or blockers expected at the surgical site (i.e., "selective spectrum" imaging). By utilizing selective spectrum imaging, the amount of time required to acquire a spectral image can be minimized such that information can be acquired in real-time or near real-time and utilized during the procedure. In various examples, the wavelengths can be selected by a clinician or by a control circuit based on clinician input. In a particular example, the wavelengths can be selected based on, for example, machine learning and / or big data accessible to the control circuit via the cloud.

[0076] The application of spectral imaging to the foregoing tissue can be utilized during surgery to measure the distance between the waveform emitter and important structures blocked by the tissue. In one aspect of the present disclosure, referring now to FIGS. 14 and 15, a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125 is shown. The time-of-flight sensor system 1104 can be incorporated into a surgical visualization system 100 (FIG. 1) in certain examples. The time-of-flight sensor system 1104 includes a waveform emitter 1106 and a waveform receiver 1108 on the same surgical device 1102. The emitted wave 1124 extends from the emitter 1106 to the important structure 1101, and the received wave 1125 is reflected by the important structure 1101 to the receiver 1108. The surgical device 1102 is positioned through a trocar 1110 extending into the patient's cavity 1107.

[0077] The waveforms 1124, 1125 are configured to penetrate the blocking tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 can be wavelengths in the NIR spectrum or the SWIR spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or a photoacoustic signal can be emitted from the emitter 1106 and can penetrate the tissue 1103 hiding the important structure 1101. The emitted waveform 1124 can be reflected by the important structure 1101. The received waveform 1125 can be delayed due to the distance d between the distal end of the surgical device 1102 and the important structure 1101. In various examples, the waveforms 1124, 1125 can be selected to target the important structure 1101 within the tissue 1103 based on the spectral signature of the important structure 1101, as further described herein. In various examples, the emitter 1106 is configured to provide an on and off binary signal, as shown for example in FIG. 15, and can be measured by the receiver 1108.

[0078] Based on the delay between the emitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d (FIG. 14). The time-of-flight timing diagram 1130 of the emitter 1106 and the receiver 1108 in FIG. 14 is shown in FIG. 15. The delay is a function of the distance d, and the distance d is given by the following formula.

[0079] [Number] Wherein, c = the speed of light, t = the length of the pulse, q1 = the charge accumulated while the light is being emitted, q2 = the charge accumulated while the light is not being emitted.

[0080] As provided herein, the time of flight of the waveforms 1124, 1125 corresponds to the distance d in FIG. 14. In various examples, additional emitters / receivers and / or pulse signals from the emitter 1106 can be configured to emit non-transmissive signals. The non-transmissive tissue can be configured to determine the distance to the surface 1105 of the tissue 1103 that is blocking the emitter. In various examples, the depth of the critical structure 1101 can be determined by the following formula: d A = d w - d t . Wherein, d A = the depth of the critical structure 1101, d w = the distance from the emitter 1106 to the critical structure 1101 (d in FIG. 14), and also, d t = the distance from the emitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the tissue 1103 that is blocking it.

[0081] In one aspect of the present disclosure, referring now to FIG. 16, a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c is shown. The time-of-flight sensor system 1204 can be incorporated into a surgical visualization system 100 (FIG. 1) in a particular example. The time-of-flight sensor system 1204 includes a waveform emitter 1206 and a waveform receiver 1208. The waveform emitter 1206 is positioned on a first surgical device 1202a, and the waveform receiver 1208 is positioned on a second surgical device 1202b. The surgical devices 1202a, 1202b are positioned through respective trocars 1210a, 1210b extending into a cavity 1207 of a patient. The emitted waves 1224a, 1224b, 1224c extend from the emitter 1206 toward the surgical site, and the received waves 1225a, 1225b, 1225c are reflected from various structures and / or surfaces at the surgical site to the receiver 1208.

[0082] The different emitted waves 1224a, 1224b, 1224c are configured to target different types of substances at the surgical site. For example, wave 1224a targets the obstructing tissue 1203, wave 1224b targets the first critical structure 1201a (e.g., a blood vessel), and wave 1224c targets the second critical structure 1201b (e.g., a cancerous tumor). The wavelengths of waves 1224a, 1224b, 1224c may be wavelengths in the visible light, NIR, or SWIR spectra. For example, visible light can reflect off the surface 1205 of the tissue 1203, and the NIR waveform and / or SWIR waveform can be configured to penetrate the surface 1205 of the tissue 1203. In various aspects, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or a photoacoustic signal can be emitted from the emitter 1206 as described herein. In various examples, waves 1224b, 1224c can be selected to target the critical structures 1201a, 1201b within the tissue 1203 based on the spectral signatures of the critical structures 1201a, 1201b as further described herein. Photoacoustic imaging is further described in various U.S. patent applications incorporated herein by reference.

[0083] The emitted waves 1224a, 1224b, 1224c can be reflected from the target substances (i.e., the surface 1205, the first critical structure 1201a, and the second structure 1201b, respectively). The received waveforms 1225a, 1225b, 1225c are delayed by the distances d shown in FIG. 16 1a , d 2a , d 3a , d 1b , d 2b , d 2c .

[0084] In a time-of-flight sensor system 1204 where the emitter 1206 and the receiver 1208 are independently positionable (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), the various distances d 1a , d 2a , d 3a , d 1b, d 2b , d 2c can be calculated from the known positions of emitter 1206 and receiver 1208. For example, when surgical devices 1202a, 1202b are robotically controlled, the positions may be known. Based on the positions of emitter 1206 and photoreceiver 1208, and knowledge regarding the time of the photon stream until a particular tissue is targeted and the information received by receiver 1208 of that particular response, distance d 1a , d 2a , d 3a , d 1b , d 2b , d 2c can be determined. In one aspect, the distance to blocked critical structures 1201a, 1201b can be triangulated using the transmission wavelength. Since the speed of light is constant for any wavelength of visible or invisible light, time-of-flight sensor system 1204 can determine various distances.

[0085] Referring further to FIG. 16, in various examples, in the images provided to the clinician, receiver 1208 can be rotated such that the centroid of the target structure in the resulting image remains constant, i.e., within a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such an orientation can quickly communicate one or more associated distances and / or viewpoints with respect to the critical structure. For example, as shown in FIG. 16, the surgical site is displayed from a viewpoint where critical structure 1201a is perpendicular to the field plane (i.e., the blood vessel is facing in and out of the page). In various examples, such an orientation may be a default setting, but the field of view can be rotated or otherwise adjusted by the clinician. In a particular example, the clinician can switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0086] In various examples, the receiver 1208 may be mounted on a trocar or cannula, such as trocar 1210b, through which the surgical device 1202b is disposed. In other examples, the receiver 1208 can be attached to a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 1208 can be mounted on a movable arm separate from the robot that controls the surgical device 1202a, or on an operating room (OR) table that can be aligned with the intraoperative robotic coordinate plane. In such examples, the positions of the emitter 1206 and the receiver 1208 can be aligned in the same coordinate plane so that the distance can be triangulated from the output of the time-of-flight sensor system 1204.

[0087] A combination of a time-of-flight sensor system and near-infrared spectroscopy (NIRS), called TOF-NIRS, which is capable of measuring the time-resolved characteristics of NIR light with nanosecond resolution, can be found in the literature "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT" in Journal of the American Society for Horticultural Science, May 2013 vol.138 no.3 225-228, which is hereby incorporated by reference in its entirety and is accessible from journal.ashspublications.org / content / 138 / 3 / 225.full.

[0088] In various examples, the time-of-flight spectral waveform is configured to determine the depth of the critical structure and / or the proximity of the surgical device to the critical structure. Further, the various surgical visualization systems disclosed herein include surface mapping logic configured to create a three-dimensional rendering on the surface of the visible tissue. In such examples, even if the visible tissue obscures the critical structure, the clinician can recognize the proximity (or lack thereof) of the surgical device to the critical structure. In one example, the topography of the surgical site is provided on a monitor by the surface mapping logic. When the critical structure is close to the surface of the tissue, spectral imaging can communicate the location of the critical structure to the clinician. For example, spectral imaging can detect structures within 5 mm or 10 mm of the surface. In other examples, spectral imaging can detect structures 10 or 20 mm below the surface of the tissue. Based on the known limitations of the spectral imaging system, the system is configured to communicate that the critical structure is out of range if it is not simply detected by the spectral imaging system. Accordingly, the clinician can continue to move the surgical device and / or continue to manipulate the tissue. When the critical structure moves within the range of the spectral imaging system, the system can identify the structure and thus communicate that the structure is within range. In such examples, a warning can be provided when the structure is first identified and / or further moves within a predefined proximity zone. In such examples, even if the critical structure is not identified by the spectral imaging system with known boundaries / ranges, proximity information (i.e., not in proximity) can be provided to the clinician.

[0089] The various surgical visualization systems disclosed herein can be configured to identify, during a surgical procedure, the presence and / or proximity of critical structures and to alert a clinician before an inadvertent incision and / or transection damages a critical structure. In various aspects, the surgical visualization system is configured to identify one or more of the following critical structures, such as the ureter, intestine, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], facial nerve branches, vagus nerve, and their branches), blood vessels (including the pulmonary and lobar arteries and veins, inferior mesenteric artery [IMA] and its branches, superior rectal artery, sigmoid artery, and left colic artery), superior mesenteric artery (SMA) and its branches (including the middle colic artery, right colic artery, ileocolic artery), hepatic artery and its branches, portal vein and its branches, splenic artery / vein and its branches, external and internal iliac vessels (lower abdomen), short gastric arteries, uterine arteries, median sacral vessels, and lymph nodes. Further, the surgical visualization system is configured to indicate the proximity of the surgical device to a critical structure and / or to alert the clinician when the surgical device approaches a critical structure.

[0090] Various aspects of the present disclosure provide identification of important structures during surgery (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking that enable visualization of important structures beneath the surface of tissue, such as beneath the surface of tissue by 1.0 - 1.5 cm. In other examples, the surgical visualization system can identify structures less than 1.0 cm or greater than 1.5 cm beneath the surface of the tissue. For example, a surgical visualization system that can identify only structures within 0.2 mm of the surface may be useful, for example, when not otherwise visible due to depth. In various aspects, the surgical visualization system can expand the clinician's field of view, for example, by virtually displaying important structures as an overlay on a visible white light image on the surface of the visible tissue. The surgical visualization system can provide real-time three-dimensional spatial tracking of the distal tip of a surgical instrument and can provide a proximity warning when the distal tip of the surgical instrument moves within a range of an important structure, such as within 1.0 cm of the important structure.

[0091] The various surgical visualization systems disclosed herein can identify when an incision is too close to a critical structure. Based on temperature (i.e., overly hot near a critical structure where there may be a risk of damaging / heating / melting the critical structure), and / or tension (i.e., overly high tension near a critical structure where there may be a risk of damaging / lacerating / pulling the critical structure), an incision may be "too close" to a critical structure. Such surgical visualization systems can facilitate incisions around blood vessels, for example, when dissecting tissue around a blood vessel prior to ligation. In various examples, a thermal imaging camera can be utilized to read the heat of the surgical site and provide a warning to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the temperature of the tool exceeds a predetermined threshold (such as 120°F), the warning can be provided to the clinician at a first distance (such as 10 mm), and if the temperature of the tool is below the established threshold, the warning can be provided to the clinician at a second distance (such as 5 mm). The established threshold and / or warning distance can be programmable by default settings and / or by the clinician. Additionally or alternatively, proximity warnings can be coupled to heat measurements performed by the tool itself, such as a thermocouple that measures heat within the distal jaw of a monopolar or bipolar cutting instrument or vascular sealer.

[0092] The various surgical visualization systems disclosed herein can provide sufficient sensitivity and specificity to critical structures to enable a clinician to proceed with a quick but safe incision with confidence, based on standards of care and / or device safety data. This system can function in real-time during surgery with a minimal risk of ionizing radiation to the patient or clinician, and in various examples, there is no risk of ionizing radiation to the patient or clinician. In contrast, in fluoroscopy, the patient and clinician can be exposed to ionizing radiation via an X-ray beam that is utilized, for example, to view anatomical structures in real-time.

[0093] The various surgical visualization systems disclosed herein can be configured to detect and identify one or more desired types of critical structures within the forward path of a surgical device, such as when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization systems can be configured to detect and identify one or more types of critical structures in, for example, the surrounding area of the surgical device and / or in multiple planes / ranges.

[0094] The various surgical visualization systems disclosed herein can be easily operated and / or interpreted. Further, the various surgical visualization systems can incorporate an "override" feature that allows a clinician to override default settings and / or operations. For example, a clinician can selectively turn off warnings from the surgical visualization system and / or approach a critical structure closer than presented by the surgical visualization system when the risk to the critical structure is lower than the risk of avoiding the area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue may be greater than the risk of damaging the critical structure).

[0095] The various surgical visualization systems disclosed herein can be incorporated into a surgical system and / or used during a surgical procedure with a limited impact on the workflow. In other words, the implementation of the surgical visualization system may not require changing the way the surgical procedure is performed. Further, the surgical visualization system can be economical compared to the cost of inadvertent dissection. Data has shown a reduction in inadvertent damage to critical structures and can facilitate an increase in reimbursement amounts.

[0096] The various surgical visualization systems disclosed herein can operate in real-time or near real-time and well in advance to enable a clinician to anticipate critical structures. For example, the surgical visualization system can provide sufficient time for "slow down, evaluate, and avoid" to maximize the efficiency of the surgical technique.

[0097] The various surgical visualization systems disclosed herein may not require a contrast agent or dye to be injected into tissue. For example, spectral imaging is configured to visualize hidden structures during surgery without using a contrast agent or dye. In other examples, the contrast agent may be easier to inject into the appropriate layer of tissue than other visualization systems. The time between injection of the contrast agent and visualization of the critical structure can be, for example, less than 2 hours.

[0098] The various surgical visualization systems disclosed herein may be coupled with clinical data and / or device data. For example, the data can provide a boundary of the distance from the tissue that the surgeon does not want to damage to the energy-activated surgical device (or other potentially damaging device). Any data module that interfaces with the surgical visualization system disclosed herein can be provided integrally or separately with the robot, for example, to enable use with a stand-alone surgical device in an open or laparoscopic procedure. The surgical visualization system can be compatible with robotic surgical systems in various examples. For example, the visualization image / information can be displayed within the robot console.

[0099] In various examples, the clinician may not be able to know the location of the critical structure relative to the surgical tool. For example, if the critical structure is embedded in tissue, the clinician may not be able to confirm the location of the critical structure. In certain examples, the clinician may wish to maintain the surgical device outside the range of positions surrounding the critical structure and / or away from the visible tissue covering the hidden critical structure. If the location of the hidden critical structure is unknown, there is a risk that the clinician may approach the critical structure too closely, resulting in inadvertent trauma and / or incision of the critical structure and / or applying excessive energy, heat, and / or tension near the critical structure. Alternatively, there may also be a risk that the clinician stays far away from a suspected critical structure location and tries to avoid the critical structure, which may affect the tissue in an undesirable location.

[0100] A surgical visualization system is provided that presents tracking of a surgical device relative to one or more critical structures. For example, the surgical visualization system can track the proximity of the surgical device to the critical structure. Such tracking can be performed during the surgery, in real-time, and / or near real-time. In various examples, the tracking data can be provided to the clinician via a display screen (e.g., a monitor) of the imaging system.

[0101] In one aspect of the present disclosure, the surgical visualization system includes a surgical device including an emitter configured to emit a structured light pattern onto a visible surface, an imaging system including a camera configured to detect an embedded structure and the structured light pattern on the visible surface, and a control circuit in signal communication with the camera and the imaging system, the control circuit being configured to determine a distance from the surgical device to the embedded structure and provide a signal indicative of the distance to the imaging system. For example, the distance can be determined based on a three-dimensional image of the illuminated structure provided by images from a plurality of lenses (e.g., a left lens and a right lens) of the camera by calculating the distance from the camera to the critical structure illuminated by fluoroscopy. The distance from the surgical device to the critical structure can be triangulated, for example, based on the known positions of the surgical device and the camera. Alternative means for determining the distance to the embedded critical structure are further described herein. For example, an NIR time-of-flight distance sensor can be used. Additionally or alternatively, the surgical visualization system can determine the distance to the visible tissue overlapping / covering the embedded critical structure. For example, the surgical visualization system can identify the hidden critical structure and extend the image of the hidden critical structure by outlining a schematic of the hidden critical structure on the visible structure, e.g., drawing a line on the surface of the visible tissue. The surgical visualization system can further determine the distance to the extended line on the visible tissue.

[0102] As provided by the various surgical visualization systems disclosed herein, by providing clinicians with up-to-date information regarding the proximity of surgical devices to hidden and / or visible structures, clinicians can make more informed decisions regarding the placement of surgical devices relative to hidden critical structures. For example, a clinician can view in real-time / during the procedure the distance between a surgical device and a critical structure, and in certain instances, an imaging system can provide alerts and / or warnings when the surgical device moves within a predefined proximity and / or zone of a critical structure. In certain instances, the alerts and / or warnings can be provided when the trajectory of the surgical device indicates a likelihood of colliding with a "no fly" zone near a critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm or more of a critical structure). In such instances, without the need for a verifying physician to monitor the location suspected to be a critical structure and the proximity of the surgical device thereto, a clinician can maintain the procedure throughout the surgery. As a result, certain surgical procedures can be performed more quickly with fewer pauses / interruptions and / or with improved accuracy and / or certainty. In one aspect, a surgical visualization system can be utilized to detect tissue diversity, e.g., diversity of tissue within an organ, in order to distinguish tumor / cancerous / diseased tissue from healthy tissue. Such a surgical visualization system can maximize the removal of diseased tissue while minimizing the removal of healthy tissue.

[0103] Surgical hub system The various visualization or imaging systems described herein can be incorporated into a surgical hub system such as those illustrated in connection with FIGS. 17-19 and described in further detail below.

[0104] Referring to FIG. 17, a computer-implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that can include a remote server 2113 connected to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 that communicates with a cloud 2104 that can include a remote server 2113. In one example, as shown in FIG. 17, a surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112 that are configured to communicate with each other and / or with the hub 2106. In some aspects, a surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers greater than or equal to 1.

[0105] FIG. 18 shows an example of a surgical system 2102 used to perform a surgical procedure on a patient lying on an operating table 2114 within an operating room 2116. The robotic system 2110 is used as part of the surgical system 2102 in a surgical procedure. The robotic system 2110 includes a surgeon's console 2118, a patient-side cart 2120 (surgical robot), and a surgical robot hub 2122. While the surgeon views the surgical site through the surgeon's console 2118, the patient-side cart 2120 can manipulate at least one removably coupled surgical tool 2117 through a minimally invasive incision in the patient's body. Images of the surgical site can be obtained by a medical imaging device 2124, which can be manipulated by the patient-side cart 2120 to orient the imaging device 2124. The robot hub 2122 can process images of the surgical site and then use the processed images for display to the surgeon via the surgeon's console 2118.

[0106] Other types of robotic systems can be easily adapted for use with the surgical system 2102. Various examples of robotic systems and surgical tools suitable for use in the present disclosure are described in various U.S. patent applications incorporated herein by reference.

[0107] Various examples of cloud-based analysis executed by the cloud 2104 and suitable for use in the present disclosure are described in various U.S. patent applications incorporated herein by reference.

[0108] In various aspects, the imaging device 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0109] The optical components of the imaging device 2124 may include one or more illumination light sources and / or one or more lenses. One or two or more illumination light sources may be directed to illuminate a portion of the surgical field. One or two or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0110] One or two or more illumination light sources can be configured to emit electromagnetic energy within the visible and invisible spectra. The visible spectrum, sometimes referred to as the optical spectrum or emission spectrum, is a portion of the electromagnetic spectrum visible to the human eye (i.e., detectable by the human eye) and is sometimes referred to as visible light, or simply light. A typical human eye responds to wavelengths of approximately 380 nm to approximately 750 nm in air.

[0111] The invisible spectrum (i.e., the non-emitting spectrum) is a part of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum and these become invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum and these become invisible ultraviolet, X-rays, and gamma ray electromagnetic radiation.

[0112] In various aspects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, choledochoscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0113] In one aspect, the imaging device uses multispectral monitoring to distinguish topography from underlying structures. A multispectral image captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments capable of sensing light at specific wavelengths beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging methods can enable the extraction of additional information that the human eye cannot capture with its red, green, and blue receptors. The use of multispectral imaging is described in various U.S. patent applications incorporated herein by reference. Multispectral monitoring can be a useful tool for performing one or more of the above-described tests on the tissue being treated and for repositioning the surgical field after a surgical task is completed.

[0114] It is self-evident that strict sterilization of the operating room and surgical equipment is necessary in any surgical procedure. The strict hygiene and sterilization conditions required in the "surgical theater", i.e., the operating room or treatment room, require the highest level of sterility for all medical devices and equipment. As part of the above sterilization process, it is necessary to sterilize anything that comes into contact with the patient or enters the sterile field (including the imaging device 2124 and its accessories and components). It will be understood that the sterile field can be regarded as a specific area considered to be free of microorganisms, such as within a tray or on a sterile towel, or the sterile field can be regarded as the area immediately surrounding the patient prepared for the surgical procedure. The sterile field can include scrubbed team members wearing appropriate clothing, as well as all supplies and fixtures within that area.

[0115] In various aspects, the visualization system 2108, as shown in FIG. 18, includes one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing units, one or more storage arrays, and one or more displays. In one aspect, the visualization system 2108 includes interfaces for HL7, PACS, and EMR. The various components of the visualization system 2108 are described in various U.S. patent applications incorporated herein by reference in the present disclosure.

[0116] As shown in FIG. 18, the primary display 2119 is positioned within the sterile field so as to be visible to the operator located on the operating table 2114. Additionally, the visualization tower 21121 is positioned outside the sterile field. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 that face away from each other. The visualization system 2108 guided by the hub 2106 is configured to regulate the information flow to the operators inside and outside the sterile field using the displays 2107, 2109, and 2119. For example, the hub 2106 can cause the snapshot of the surgical site recorded by the imaging device 2124 to be displayed on the non-sterile display 2107 or 2109 while maintaining the live video of the surgical site on the primary display 2119 for the visualization system 2108. The snapshot on the non-sterile display 2107 or 2109 can enable, for example, a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0117] In one aspect, the hub 2106 is also configured to send the diagnostic input or feedback entered by a non-sterile operator in the visualization tower 21121 to the primary display 2119 within the sterile area so that it can be seen by the sterile operator located on the operating table. In one example, the input may be in the form of a modification to the snapshot displayed on the non-sterile display 2107 or 2109 that can be sent by the hub 2106 to the primary display 2119.

[0118] Referring to FIG. 18, surgical instrument 2112 is used as part of surgical system 2102 in a surgical procedure. Hub 2106 is also configured to condition the flow of information to the display of surgical instrument 2112 as described in various U.S. patent applications incorporated herein by reference in the present disclosure. Diagnostic inputs or feedback entered by a non-sterile operator at visualization tower 21121 can be sent by hub 2106 to surgical instrument display 2115 within the sterile field where the diagnostic inputs or feedback can be viewed by the operator of surgical instrument 2112. Exemplary surgical instruments suitable for use with surgical system 2102 are described in various U.S. patent applications incorporated herein by reference in the present disclosure.

[0119] FIG. 19 shows a computer-implemented interactive surgical system 2200. The computer-implemented interactive surgical system 2200 is similar to the computer-implemented interactive surgical system 2100 in many respects. The surgical system 2200 includes at least one surgical hub 2236 that communicates with a cloud 2204 that may include a remote server 2213. In one aspect, the computer-implemented interactive surgical system 2200 includes a surgical hub 2236 connected to a plurality of operating room devices such as, for example, intelligent surgical instruments, robots, and other computerized devices located within the operating room. The surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. As shown in the embodiment of FIG. 19, the surgical hub 2236 is connected to an imaging module 2238 connected to an endoscope 2239, a generator module 2240 connected to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communication module 2230, a processor module 2232, a storage array 2234, a smart device / instrument 2235 optionally connected to a display 2237, and a non-contact sensor module 2242. The operating room devices are connected to cloud computing resources and data storage via the surgical hub 2236. A robot hub 2222 may also be connected to the surgical hub 2236 and cloud computing resources. In particular, the device / instrument 2235, visualization system 2209 may be connected to the surgical hub 2236 via a wired or wireless communication standard or protocol as described herein. The surgical hub 2236 may be connected to a hub display 2215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display, and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with the images and overlay images.

[0120] Situation recognition Various visualization systems or aspects of visualization systems described herein can be utilized as part of a situation awareness system that can be embodied or executed by surgical hubs 2106, 2236 (Figs. 17 - 19). In particular, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their position, orientation, and movement), tissue, structures, the user, and other things located within the surgical field or operating room can provide context data that can be utilized by the situation awareness system for purposes such as inferring the type of surgical procedure being performed or its steps, the type of tissue and / or structures being manipulated by the surgeon. This context data can then be utilized by the situation awareness system to provide warnings to the user, propose subsequent steps or actions for the user to perform, prepare surgical devices in anticipation of the user's use (e.g., activate an electrosurgical generator in anticipation of the use of an electrosurgical instrument in a subsequent step of a surgical procedure), and intelligently control surgical instruments (e.g., customize the operating parameters of a surgical instrument based on the specific health profile of each patient), among other things.

[0121] An "intelligent" device that includes a control algorithm that responds to sensed data may be an improvement over a "dumb" device that operates without considering the sensed data. However, some sensed data may be incomplete or inconclusive when considered alone, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the procedure context (e.g., knowing the type of tissue being operated on or the type of procedure being performed), the control algorithm may inaccurately or sub-optimally control the modular device when given sensed data that does not include a specific context. The modular device can include any surgical device that can be controlled by a situation awareness system, such as a visualization system device (e.g., a camera or display screen), a surgical instrument (e.g., an ultrasonic surgical instrument, an electrosurgical instrument, or a surgical stapler), and other surgical devices (e.g., a smoke evacuator). For example, the optimal way to control a surgical instrument in response to a particular sensed parameter may vary depending on the specific type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., resistance to tearing) and thus respond differently to actions taken by the surgical instrument. Therefore, even if the same measurement is sensed for a particular parameter, it may be desirable for the surgical instrument to take different actions. As one specific example, the optimal way to control a surgical stapling and cutting instrument in response to sensing an unexpectedly high force to close its end effector may vary depending on whether the tissue type is susceptible to tearing or is resistant to it. In the case of tissue that is susceptible to tearing, such as lung tissue, the control algorithm for the instrument optimally ramps down the motor in response to the unexpectedly high force to close in order to avoid tissue tearing. In the case of tissue that is resistant to tearing, such as stomach tissue, the control algorithm for the instrument optimally ramps up the motor in response to the unexpectedly high force to close in order to ensure that the end effector is properly clamped to the tissue.If it is not known whether the lung tissue or the stomach tissue is clamped, the control algorithm may make a sub-optimal decision.

[0122] One solution utilizes a surgical hub that includes a system configured to derive information about a surgical procedure being performed based on data received from various data sources and then appropriately control a paired modular device. In other words, the surgical hub is configured to infer information about the surgical procedure from the received data and then control the modular device paired with the surgical hub based on the inferred context of the surgical procedure. FIG. 20 shows a diagram of a situation-aware surgical system 2400 according to at least one aspect of the present disclosure. In some examples, data source 2426 may include, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database including patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0123] Surgical hub 2404 may be similar to hub 106 in many respects and may be configured to derive context information about the surgical procedure from the data, for example, based on a particular combination of received data or the particular order in which data is received from data source 2426. The context information inferred from the received data may include, for example, the type of surgical procedure being performed, a particular step of the surgical procedure that the surgeon is performing, the type of tissue being operated on, or the body cavity that is the subject of the procedure. This ability of surgical hub 2404, according to some aspects, to derive or infer information related to the surgical procedure from the received data may be referred to as "situation awareness." In one example, surgical hub 2404 can incorporate a situation awareness system that is hardware and / or programming associated with surgical hub 2404 that derives context information related to the surgical procedure from the received data.

[0124] The situation recognition system of the surgical hub 2404 can be configured to derive context information from data received from various different data sources 2426. In one example, the situation recognition system correlates various inputs (e.g., data from the database 2422, the patient monitoring device 2424, and / or the modular device 2402) with corresponding context information regarding the surgical procedure using a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained with training data. In other words, the machine learning system can be trained to accurately derive context information regarding the surgical procedure from the provided inputs. In another example, the situation recognition system can include a look-up table that stores pre-characterized context information regarding the surgical procedure in correspondence with one or more inputs (or ranges of inputs) corresponding to the context information. In response to a query by one or more inputs, the look-up table can return the corresponding context information of the situation recognition system to control the modular device 2402. In one example, the context information received by the situation recognition system of the surgical hub 2404 is associated with a specific control adjustment or a series of control adjustments of one or more modular devices 2402. In another example, the situation recognition system includes a further machine learning system, a look-up table, or other such system that generates or retrieves one or more control adjustments of one or more modular devices 2402 when context information is provided as an input.

[0125] The surgical hub 2404 incorporating the situation awareness system provides many benefits to the surgical system 2400. One benefit includes improving the interpretation of sensed and collected data, which in turn improves the processing accuracy and / or use of data during a surgical procedure. To return to a previous example, the situation-aware surgical hub 2404 can determine which type of tissue is being operated on, and thus, if an unexpectedly high force is detected to close the end effector of a surgical instrument, the situation-aware surgical hub 2404 can correctly ramp up or ramp down the motor of the surgical instrument according to the type of tissue.

[0126] As another example, the type of tissue being operated on can affect the adjustments made to the compression speed and load threshold of a surgical stapling and cutting instrument for specific tissue gap measurements. The situation-aware surgical hub 2404 can estimate whether the surgical procedure being performed is a thoracic procedure or an abdominal procedure, whereby the surgical hub 2404 can determine whether the tissue clamped by the end effector of the surgical stapling and cutting instrument is a lung (in the case of thoracic surgery) or a stomach (in the case of abdominal surgery). The surgical hub 2404 can then appropriately adjust the compression speed and load threshold of the surgical stapling and cutting instrument according to the type of tissue.

[0127] As yet another example, the type of body cavity being operated on during an insufflation procedure can affect the function of the smoke evacuator. The situation-aware surgical hub 2404 can determine whether the surgical site is under pressure (by determining that the surgical procedure utilizes insufflation), and can determine the type of procedure. Since the type of procedure is generally performed within a specific body cavity, the surgical hub 2404 can appropriately control the motor speed of the smoke evacuator according to the body cavity being operated on. Thus, the situation-aware surgical hub 2404 can provide a certain amount of smoke evacuation for both thoracic and abdominal surgeries.

[0128] As yet another example, the type of procedure being performed can affect the energy level that is optimal for an ultrasonic surgical instrument or a radiofrequency (RF) electrosurgical instrument to operate. For example, arthroscopic procedures require higher energy levels because the end effector of an ultrasonic surgical instrument or an RF electrosurgical instrument is immersed in fluid. The situation-aware surgical hub 2404 can determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 can then adjust the RF power level or the ultrasonic amplitude (i.e., the "energy level") of the generator to compensate for the fluid-filled environment. In connection therewith, the type of tissue being operated on can affect the energy level that is optimal for an ultrasonic surgical instrument or an RF electrosurgical instrument to operate. The situation-aware surgical hub 2404 can determine which type of surgical procedure is being performed and then customize the energy levels of the ultrasonic surgical instrument or the RF electrosurgical instrument, respectively, according to the expected tissue profile for the surgical procedure. Further, the situation-aware surgical hub 2404 can be configured to adjust the energy levels of the ultrasonic surgical instrument or the RF electrosurgical instrument not only for each procedure but also over the course of the surgical procedure. The situation-aware surgical hub 2404 can determine which step of the surgical procedure is being performed or will be performed subsequently and then update the control algorithms of the generator and / or the ultrasonic surgical instrument or the RF electrosurgical instrument to set the energy level to a value appropriate for the tissue type expected according to the step of the surgical procedure.

[0129] As yet another example, additional data sources 2426 may be drawn upon to improve the conclusions drawn by surgical hub 2404 from one data source 2426. The situation-aware surgical hub 2404 may enhance the data received from the modular device 2402 with context information constructed from other data sources 2426 regarding the surgical procedure. For example, the situation-aware surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding has stopped at the surgical site) according to video or image data received from a medical imaging device. However, in some cases, the video or image data may not be conclusive. Thus, in one illustration, the surgical hub 2404 may be further configured to compare a physiological measurement (e.g., blood pressure sensed by a BP monitor communicatively coupled to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (FIG. 2) communicatively coupled to the surgical hub 2404) to make a determination regarding the integrity of a staple line or tissue weld. In other words, the situation-aware system of the surgical hub 2404 can provide additional context when analyzing visualization data in consideration of physiological measurement data. The additional context can be useful when the visualization data may not be conclusive or complete on its own.

[0130] As another benefit, the paired modular device 2402 may be actively and automatically controlled according to specific steps of the surgical procedure being performed to reduce the number of times healthcare providers need to interact with or control the surgical system 2400 during the surgical procedure. For example, the situation-aware surgical hub 2404 may actively activate a generator to which an RF electrosurgical instrument is connected when it determines that subsequent steps of the procedure require the use of the instrument. By actively activating the energy source, the instrument can be made ready for use as soon as the preceding steps of the procedure are completed.

[0131] As another example, the situation-aware surgical hub 2404 can determine, according to features at the surgical site where it is anticipated that the surgeon will need to see whether the current or subsequent steps of a surgical procedure require different fields of view or degrees of magnification on the display. The surgical hub 2404 can then appropriately and actively change the displayed field of view (e.g., supplied by a medical imaging device for the visualization system 108), such that the display will automatically adjust throughout the surgical procedure.

[0132] As yet another example, the situation-aware surgical hub 2404 can determine which step of a surgical procedure is being performed or will be performed thereafter, and whether a particular data or comparison between data is required for that step of the surgical procedure. The surgical hub 2404 can be configured to automatically call up a data screen based on the step of the surgical procedure being performed, without waiting for the surgeon to request specific information.

[0133] Another benefit may include checking for errors during the setup of a surgical procedure or during the surgical procedure itself. For example, the situation-aware surgical hub 2404 can determine whether the surgical site is properly or optimally set up for the surgical procedure to be performed. The surgical hub 2404 can determine the type of surgical procedure being performed and read out the corresponding checklist, product locations, or setup needs (e.g., from memory), and then be configured to compare the current layout of the surgical site to the standard layout for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 can be configured to compare, for example, a list of items for a procedure scanned by a suitable scanner and / or a list of devices paired with the surgical hub 2404 to the recommended or expected manifest of items and / or devices for a given surgical procedure. If there are discontinuities between the lists, the surgical hub 2404 can be configured to provide a warning indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing. In one example, the surgical hub 2404 can be configured to determine, for example, the relative distance or position of the modular device 2402 and the patient monitoring device 2424 by means of a proximity sensor. The surgical hub 2404 can compare the relative position of the devices to the recommended or expected layout for a particular surgical procedure. If there are discontinuities between the layouts, the surgical hub 2404 can be configured to provide a warning indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0134] As another example, the situation-aware surgical hub 2404 can determine whether a surgeon (or other healthcare provider) is making a mistake or deviating in some other way from the series of actions required during a surgical procedure. For example, the surgical hub 2404 can determine the type of surgical procedure being performed, read a corresponding list of steps or order of equipment use (e.g., from memory), and then compare the steps being performed or equipment being used during the surgical procedure to the expected steps or equipment for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one illustration, the surgical hub 2404 can be configured to provide a warning indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0135] Overall, the situation awareness system for the surgical hub 2404 improves the outcome of a surgical procedure by adjusting (e.g., tuning for different tissue types) surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure and verifying actions during the surgical procedure. The situation awareness system also improves the efficiency of the surgeon during the performance of a surgical procedure by automatically suggesting the next step, providing data, and adjusting displays and other modular devices 2402 within the operating field according to the specific context of the procedure.

[0136] Referring now to FIG. 21, for example, a timeline 2500 showing the situation awareness of a hub such as surgical hub 106 or 206 (FIGS. 1-11) is shown. The timeline 2500 shows an exemplary surgical procedure and context information that can be derived from data received by the surgical hubs 106, 206 from data sources at each step of the surgical procedure. The timeline 2500 shows the typical steps that a nurse, surgeon, and other healthcare personnel would take during a lobectomy procedure that begins with setting up the operating room and ends with transferring the patient to the post-operative recovery room.

[0137] The situation awareness surgical hubs 106, 206 receive data from a data source that includes data generated each time a healthcare provider uses a modular device paired with the surgical hubs 106, 206 over the course of a surgical procedure. The surgical hubs 106, 206 receive this data from the paired modular devices and other data sources. And when new data is received, such as which step of the procedure is being performed at any given time, an estimate regarding the ongoing procedure (i.e., context information) can be continuously derived. The situation awareness system of the surgical hubs 106, 206 can, for example, record data regarding the procedure to generate a report, verify the steps being taken by the healthcare provider, provide data or prompts that may be associated with a particular procedure step (e.g., via a display screen), adjust the modular device based on the context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or an RF electrosurgical instrument), and perform any other such actions as described above.

[0138] As a first step 2502 in this exemplary procedure, a hospital staff member reads the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hubs 106, 206 determine that the procedure to be performed is a thoracic procedure.

[0139] In a second step 2504, the staff member scans the incoming medical supplies for the procedure. The surgical hubs 106, 206 cross-reference the scanned supplies with a list of supplies used in various types of procedures and confirm that the combination of supplies is appropriate for a thoracic procedure. Further, the surgical hubs 106, 206 can also determine that the procedure is not a wedge procedure (either because the incoming supplies do not include the specific supplies required for a thoracic wedge procedure or are otherwise not compatible with a thoracic wedge procedure).

[0140] In the third step 2506, the healthcare provider scans the patient's band via a scanner communicatively connected to the surgical hubs 106, 206. Subsequently, the surgical hubs 106, 206 can confirm the patient's identification information based on the scanned data.

[0141] In the fourth step 2508, the medical staff turns on the auxiliary device. The auxiliary devices used can vary according to the type of surgical procedure and the techniques used by the surgeon. In this exemplary case, these include a smoke evacuator, an inhaler, and a medical imaging device. When the auxiliary device is activated, the modular device, as an auxiliary device, can automatically pair with the surgical hubs 106, 206 located within a specific vicinity of the modular device as part of its initialization process. Subsequently, the surgical hubs 106, 206 can derive context information regarding the surgical procedure by detecting the type of modular device paired with it during this preoperative or initialization phase. In this specific example, the surgical hubs 106, 206 determine that the surgical procedure is a VATS surgery based on this specific combination of paired modular devices. Based on a combination of data from the patient's EMR, a list of medical supplies used in the surgery, and the type of modular device connected to the hub, the surgical hubs 106, 206 can generally estimate the specific procedure being performed by the surgical team. When the surgical hubs 106, 206 recognize what specific procedure is being performed, subsequently, the surgical hubs 106, 206 can read the steps of the procedure from memory or from the cloud and then cross-reference the data subsequently received from the connected data sources (e.g., modular devices and patient monitoring devices) to estimate which step of the surgical procedure the surgical team is performing.

[0142] In the fifth step 2510, the staff attaches the EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices can be paired with the surgical hubs 106, 206. When the surgical hubs 106, 206 start receiving data from the patient monitoring devices, the surgical hubs 106, 206 confirm that the patient is in the operating room.

[0143] In the sixth step 2512, the medical staff anesthetizes the patient. The surgical hubs 106, 206 can estimate that the patient is under anesthesia based on data from the modular device and / or patient monitoring device, including, for example, EKG data, blood pressure data, ventilator data, or combinations thereof. When the sixth step 2512 is completed, the preoperative part of the pulmonary resection procedure is completed and the surgical part begins.

[0144] In the seventh step 2514, the lung of the patient being operated on is deflated (while ventilation is switched to the contralateral lung). The surgical hubs 106, 206 can estimate, for example, from the ventilator data that the patient's lung has been deflated. Since the surgical hubs 106, 206 can compare detecting that the patient's lung has been deflated with the expected steps of the procedure (which can be accessed or read in advance), it can be determined that the surgical part of the procedure has already started, and thereby it can be determined that deflating the lung is the first surgical step in this particular procedure.

[0145] In the eighth step 2516, a medical imaging device (e.g., a scope) is inserted and video imagery from the medical imaging device is started. The surgical hubs 106, 206 receive medical imaging device data (i.e., video or image data) through the connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hubs 106, 206 can determine that the laparoscopic portion of the surgical procedure has started. Further, the surgical hubs 106, 206 can determine that the particular procedure being performed is a segmentectomy as opposed to a lobectomy (note that based on the data received in the second step 2504 of the procedure, a wedge procedure has already been determined by the surgical hubs 106, 206 to not be a possibility). Data from the medical imaging device 124 (FIG. 2) may be used in many different ways to determine context information regarding the type of procedure being performed. For example, it may be used to determine the angle at which the medical imaging device is oriented with respect to a visualization of the patient's anatomical structure, to monitor the number of medical imaging devices being used (i.e., powered on and paired with the surgical hubs 106, 206), or to monitor the type of visualization device being used. For example, one technique for performing a VATS lobectomy places the camera above the diaphragm at the anteroinferior corner of the patient's chest cavity, while one technique for performing a VATS segmentectomy places the camera at an intercostal position anterior to the segmental fissure. For example, using pattern recognition or machine learning techniques, a situation recognition system can be trained to recognize the position of the medical imaging device based on a visualization of the patient's anatomical structure. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing a VATS segmentectomy uses an infrared light source (which can be communicably coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in a VATS lobectomy.By tracking any or all of this data from the medical imaging device, the surgical hubs 106, 206 can determine the specific type of surgical procedure being performed and / or the technique being used for the specific type of surgical procedure.

[0146] In the ninth step 2518, the surgical team begins the incision step of the procedure. Since the surgical hubs 106, 206 receive data from the RF or ultrasonic generator indicating that the energy instrument is being fired, it can be inferred that the surgeon is in the process of incising and mobilizing the patient's lung. The surgical hubs 106, 206 can cross-reference the received data with the read steps of the surgical procedure to determine that the energy instrument being fired at this point in the process (i.e., after the above-described steps of the procedure are completed) corresponds to the incision step. In a particular example, the energy instrument can be an energy tool attached to the robotic arm of a robotic surgical system.

[0147] In the tenth step 2520, the surgical team proceeds to the ligation step of the procedure. Since the surgical hubs 106, 206 receive data from the surgical stapling and cutting instrument indicating that the instrument is being fired, it can be inferred that the surgeon is currently ligating the arteries and veins. Similar to the previous step, the surgical hubs 106, 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and cutting instrument with the steps within the read process. In a particular example, the surgical instrument can be a surgical tool attached to the robotic arm of a robotic surgical system.

[0148] In the 11th step 2522, a regional resection of the treatment is performed. The surgical hubs 106, 206 can be estimated by the surgeon to be transecting substantial tissue based on data from the surgical stapling and cutting instrument, including data from its cartridge. The cartridge data can correspond, for example, to the size or type of staples fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transected. In this case, the type of staple fired is used for substantial tissue (or other similar tissue types), whereby the surgical hubs 106, 206 can be estimated to be performing a regional resection of the treatment.

[0149] Subsequently, in the 12th step 2524, a node dissection step is performed. The surgical hubs 106, 206 can be estimated by the surgical team to be dissecting the node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. In this particular treatment, the RF or ultrasonic instrument used after the substantial tissue has been transected corresponds to the node dissection step, whereby the surgical hubs 106, 206 are able to make the above-mentioned estimation. Since different instruments are better suited for specific tasks, the surgeon should be aware of periodically switching between the surgical stapling / cutting instrument and the surgical energy (i.e., RF or ultrasonic) instrument in accordance with the specific steps during the treatment. Therefore, the specific sequence in which the stapling / cutting instrument and the surgical energy instrument are used can indicate which step of the treatment the surgeon is performing. Furthermore, in certain examples, robotic tools can be used for one or more steps during the surgical treatment, and / or handheld surgical instruments can be used for one or more steps during the surgical treatment. The surgeon(s) can, for example, use the robotic tool and the handheld surgical instrument alternately and / or use the devices simultaneously. When the 12th step 2524 is completed, the incision is closed and the postoperative part of the treatment begins.

[0150] In the 13th step 2526, the operation of awakening the patient from anesthesia is performed. The surgical hubs 106, 206 can be estimated to be awakening from anesthesia, for example, based on the ventilator data (i.e., the patient's breathing rate begins to increase).

[0151] Finally, the 14th step 2528 is for the medical staff to remove various patient monitoring devices from the patient. Therefore, the surgical hubs 2106, 2236 can be estimated that the patient has been transferred to the recovery room when the hub loses EKG, BP, and other data from the patient monitoring device. As can be seen from the description of this exemplary procedure, based on the data received from various data sources communicably coupled to the surgical hubs 2106, 2236, the surgical hubs 2106, 2236 can determine or estimate when each step of a given surgical procedure is being performed.

[0152] Situation awareness is further described in various U.S. patent applications incorporated herein by reference, which are incorporated herein by reference in their entirety. In certain examples, for example, the operation of a robotic surgical system, including various robotic surgical systems disclosed herein, may be controlled by the hubs 2106, 2236 based on that situation awareness and / or feedback from its components, and / or based on information from the cloud 2104 (FIG. 17).

[0153] Imaging system using fused images One problem specific to surgical procedures in which a surgeon relies on the imaging system 142 (FIG. 2) is an obstruction to the camera 144 (FIG. 2) that degrades the imaging system 142's ability to visualize the surgical site and, consequently, the surgeon's ability to perform the surgical tasks required for the procedure. Obstructions can include, for example, fluids (e.g., blood) present on the lens of the camera 144, on the surface of the body cavity, or otherwise at the surgical site, smoke generated by an electrosurgical instrument or other aerosols present within the body cavity, and / or tissue or other structures that overlap the target tissue or structure. The surgical system can be configured to utilize various imaging techniques for compensating for the obstruction, including multispectral imaging of sub-regions of the FOV of the camera 144, interpolation of sub-regions of previous image frames captured by the camera 144, comparative multispectral analysis of the captured images, and the like.

[0154] In one general aspect, the present disclosure relates to a surgical system configured to utilize segments of an image captured at a sampling rate via a multi - spectral light source of an imaging system 142a to minimize obstruction of visualization by various obstructions (e.g., surgical smoke). In one aspect, the surgical system can be configured to combine hyperspectral imaging with visible - light imaging to minimize image interference caused by obstructions. The surgical system can be configured to detect aspects of a surgical instrument, a surgical site, or surrounding substrate or obscured portions, for example, by utilizing distinct wavelengths or wavelength ranges of EMR. For example, the surgical system can utilize frames from a continuous scanning device to transmit distinct wavelengths of EMR, a hyperspectral imaging device configured to scan both inside and outside of the visible - light spectrum, or a second imaging system configured to emit EMR at a different length than a first or primary imaging system. Thus, the surgical system can be configured to identify obscured portions of an image frame at a particular wavelength or set of wavelengths and to interpolate or replace portions of the obscured image portion with non - obscured image portions of an image frame acquired at other EMR wavelengths to provide the user with a fully - visualized, unobstructed image of the surgical site.

[0155] In an aspect of using an imaging system including a hyperspectral imaging device, the hyperspectral imaging device can scan at a specific speed (e.g., 240 frames per second) that enables a portion of the emitted scan to include EMR from a near-infrared or UV laser source. Since the EMR at these wavelengths is not affected by obstacles such as surgical smoke and fluids in the same way as visible light, the hyperspectral imaging device can be used to obtain images of shapes, contours, or features that exist in both the hyperspectral image and the corresponding visible light image. Then, a control system of a surgical system, such as the control system 133 shown in FIG. 2, can be configured to replace the obscured portions of the image obtained using visible light with the corresponding detected hyperspectral features or image portions to complete visualization for the surgeon. As another example, the imaging system can include an adjustable EMR source (e.g., the spectral light source 150), and the adjustable EMR source can be controlled by the control system 133 to emit EMR at a wavelength or set of wavelengths (e.g., within the visible blue-green wavelength range) that minimizes the absorption of EMR by water because shielding by water or aqueous fluids is particularly likely to occur during a surgical procedure. As another example, the surgical system can further include a second imaging system in addition to the first primary imaging system (e.g., the imaging system 142 shown in FIG. 2). In this example, the first imaging system 142 can be configured for imaging in the visible or near-visible EMR spectrum, and the second imaging system can be configured for imaging in a different wavelength spectrum (e.g., long-wave IR (LWIR)). Thus, the second imaging system can be actuated or otherwise utilized by the surgical system as needed when the first imaging system is obscured. In these various aspects, the surgical system minimizes the amount of cleaning required for the camera 144 (e.g., to remove obstacles from the image sensor 135 or other scanning array) and prevents a temporary loss of visibility of the surgical field due to obstacles (e.g., surgical smoke or insufflation gas) between the camera 144 and the surgical field.

[0156] In certain examples, an imaging or visualization system is described as including a hyperspectral imaging device or as utilizing hyperspectral imaging technology. However, it should be noted that hyperspectral imaging is a particular type of multispectral imaging. In hyperspectral imaging, the wavelength “bins” are continuous, so hyperspectral imaging technology utilizes the entire EMR spectrum. In contrast, multispectral can mean that the “bins” are separated. In other words, a multispectral imaging system can sense EMR within, for example, the visible, mid-wave IR (MWIR), and LWIR portions of the EMR spectrum (e.g., there may be gaps within the near-infrared (NIR) portion of the EMR spectrum and / or between the MWIR portion and the LWIR portion that a multispectral imaging system does not sense). The imaging or visualization systems and methods described herein should not be construed as limited to any particular example, including examples that describe hyperspectral imaging. Indeed, the imaging or visualization systems and methods can broadly utilize any multispectral imaging device and technology.

[0157] To aid in the understanding of the foregoing systems and methods, various examples are described within the context of video-assisted thoracoscopic surgery (VATS) procedures. However, this is for illustrative purposes only, and it should be understood that the described systems and methods are applicable to other situations and / or surgical procedures. A VATS procedure is a surgical procedure in which one or more surgical instruments and one or more thoracoscopes (i.e., cameras) are inserted into a patient's chest cavity through slits positioned between the patient's ribs. The camera is utilized to provide the surgeon with a view of the interior of the patient's chest cavity, enabling the surgeon to properly position / move the surgical instruments and manipulate the tissues / structures within the chest cavity. Accordingly, FIG. 22 is a diagram of a surgical system 3000 during the performance of a surgical procedure on a lung 3010 according to at least one aspect of the present disclosure. A surgical system 3000 for performing video-assisted surgical procedures can include various different surgical devices, including an imaging device 3002, a grasper 3004, an electrosurgical instrument 3006, or another surgical instrument, and a smoke evacuator 3008. Additionally, the surgical system 3000 can include or be coupled to a surgical hub 2106, 2236 (FIGS. 17-19), a visualization system 2108 (FIGS. 17-19), or an imaging system 142 (FIG. 2), a control system 133 (FIG. 2), a robotic system 2110 (FIGS. 17-19), and any other system or device described herein. The imaging device 3002 can include a camera 144 (FIG. 2), a spectral light source 150 (FIG. 2), a structured light source 152 (FIG. 2), any other imaging emitter or receiver described herein, or a combination thereof. The imaging device 3002 can be configured to capture an image or video of the surgical site within the FOV 3020 and provide it to a display screen (e.g., a display 146 as in FIG. 2) for viewing by a user (e.g., a surgeon). The imaging device 3002 can be configured to sense EMR within or outside the visible light portion of the EMR spectrum and thereby visualize tissues and / or structures that are either visible or invisible to the naked eye. Based on the visualization provided by the imaging system 142 associated with the imaging device 3002, the surgeon can then control the surgical devices to manipulate the tissues and / or structures to perform the surgical procedure.

[0158] During a surgical procedure, various obscuring objects such as surgical plume 3014 or other aerosols, fluids, gases, tissues, structures, etc. may move across the FOV 3020 of the imaging device 3002, thereby preventing the imaging system 132 from fully visualizing the surgical procedure site, which in turn may adversely affect the surgeon's ability to perform the procedure. Many surgical systems 3000 include a smoke evacuator 3008 for removing surgical plume 3014, other aerosols, and gases from the body cavity during surgery. However, the smoke evacuator 3008 may not be sufficient to remove all obscuring objects, or there may be a delay associated with removing obscuring objects that prevent the surgeon from properly visualizing the surgical site. Accordingly, there is a need for systems and methods to compensate for the presence of obscuring objects and enable visualization of the surgical site through those obscuring objects.

[0159] In one aspect, an imaging system, such as imaging system 142 shown in FIG. 2, can be configured to utilize hyperspectral imaging and image fusion techniques to enable visualization through obscurants. For example, FIG. 23 is a diagram of an imaging device 3002 facing a plurality of obscurants. In this example, the target of the surgical procedure is a subsurface tumor 3038. However, in order to actually visualize the tumor 3038, the imaging device 3002 needs to compensate for several different obscurants, including fluid 3030 present on the lens of the imaging device 3002, surgical smoke 3032 present in the body cavity, blood 3034 on the surface of tissue 3036, the tissue 3036 itself, and structure 3040 located throughout the tissue 3036. In one aspect, the imaging device 3002 can be a hyperspectral imaging device configured to sense EMR across a wavelength spectrum. EMR interacts differently with various objects at different wavelengths. In particular, certain wavelengths of EMR may not be absorbed by a particular obscurant at a particular wavelength or wavelength range. Thus, by sensing EMR in multiple portions of the EMR spectrum, the imaging system 142 can visualize through the obscurants by sensing EMR at wavelengths that are not absorbed by the obscurants. Further, the wavelengths sensed by the imaging device 3002 can be selected to sense wavelengths that do not interact (or substantially do not interact) with typical or expected obscurants. In the illustrated example, the imaging device 3002 can be configured to sense EMR within the visible light, MWIR, and LWIR portions of the EMR spectrum.

[0160] In one aspect, the control system utilizes multispectral (e.g., hyperspectral) imaging to visualize the surgical site in multiple portions of the EMR spectrum and then replaces occluded portions of an image captured in one wavelength range with corresponding portions of an image captured in another wavelength range that is not absorbed by the occluder to provide the user with visualization without occlusions. An example of such an algorithm is shown in FIG. 24, which is a logical flow diagram of a process 3050 for generating a fused image using a multispectral EMR source. See also FIGS. 2 and 23 in the following description of process 3050. Process 3050, when executed by control circuit 132, can be embodied as, for example, instructions stored in a memory 134 coupled to control circuit 132 that cause control circuit 132 to execute the enumerated steps of process 3050. For simplicity, process 3050 is described as being executed by control circuit 132, but it should be understood that process 3050 can be executed by other combinations of hardware, software, and / or firmware.

[0161] Accordingly, control circuit 132, which executes process 3050, can cause imaging system 142 to sense EMR in a first wavelength range (e.g., visible light) from the surgical site (e.g., via imaging device 3002) at 3052 and then generate a corresponding first image therefrom at 3054. Correspondingly, control circuit 132 can cause imaging system 142 to sense EMR in a second wavelength range (e.g., MWIR or LWIR) from the surgical site (e.g., via imaging device 3002) at 3056 and then generate a corresponding second image therefrom at 3058.

[0162] Therefore, the control circuit 132 can determine 3060 whether the first image is at least partially blocked. The control circuit 132 can be configured to make this determination by detecting obstacles using object recognition and other computer vision techniques. If the first image is not at least partially blocked, process 3050 proceeds along the no branch, and the control circuit 132 can perform sensing 3052, 3054 EMR, and generation 3054, 3058 of corresponding images as described above. If the first image is at least partially blocked (i.e., there is an obstacle in the image), process 3050 proceeds along the yes branch, and the control circuit 132 can generate 3062 a third image by replacing the blocked portion of the first image with the corresponding portion of the second image. If the second wavelength range is selected such that it is not absorbed by what it blocks, the corresponding portion of the second image will not be blocked. Therefore, the third image will provide an unobstructed visualization of the surgical site for the surgeon to view.

[0163] For simplicity, process 3050 is described in the context of generating and combining two images captured in two different wavelength ranges, but imaging system 142 can be configured to sense and generate images in any number of wavelength ranges. FIG. 23 shows, for example, an implementation that generates an image obtained by combining image data from at least three different EMR wavelength ranges. Each of the illustrated first image 3042a, second image 3042b, third image 3042c, and fourth image 3042d includes an array of pixels 3043 that collectively visualize the surgical site in the corresponding EMR wavelength range. In this example, the first image 3042a is captured using the visible light portion of the EMR spectrum and includes a first unobstructed portion 3044a, and the remaining portion of the image 3042a is obstructed. The second image 3042b is captured using the MWIR portion of the EMR spectrum and includes a second unobstructed portion 3044b. The third image 3042c is captured using the LWIR portion of the EMR spectrum and includes a third unobstructed portion 3044c. Control system 133 can also be configured to perform various image processing techniques on the various generated images to improve the visualization provided thereby. For example, the fourth image 3042d is also captured using the visible light portion of the EMR spectrum and can thus correspond to the first image 3042a, but includes additional image processing to identify the portion 3044d obstructed by fluid (water). Accordingly, the corresponding portion of the first image 3042a can be filtered at the corresponding wavelength or wavelength range (e.g., the blue-green portion of the visible light spectrum) to remove the obstruction. Thus, control circuit 132, which executes process 3050, can be configured to generate a combined or fused image 3070 from the aforementioned initial images 3042a, 3042b, 3042c, 3042d.The fused image 3070 can include a first portion 3072 corresponding to an unobscured portion 3044a of a first image 3042a generated from the visible light portion of the EMR spectrum, a second portion 3074 corresponding to an unobscured portion 3044b of a second image 3042b generated from the MWIR portion of the EMR spectrum, a third portion 3076 corresponding to an unobscured portion 3044c of a third image 3042c generated from the LWIR portion of the EMR spectrum, and a fourth portion 3078 corresponding to an obscured portion 3044d of an image generated from the visible light portion of the EMR spectrum but post-processed to remove the blue-green portion of the visible light spectrum. Each of the aforementioned image portions 3072, 3074, 3076, 3078 can be fused together by the control system 133 to generate a fused image 3070 that provides an unobscured visualization of the tumor 3038 and any other relevant structures 3040.

[0164] Another technique that can be used to compensate for occlusions present at the surgical site is to image sub-region interpolation, whereby portions of an image that are occluded, damaged, or otherwise interfered with can be replaced by corresponding portions of images from a synchronized image set. For example, a surgical control system can utilize lucky region fusion (LRF) technology to enhance the quality of visualization provided to a user by using a plurality of image frames. In one aspect, the control system can be configured to provide the user with an occlusion-free visualization by replacing occluded portions of an image with non-occluded portions of a previously captured image. An example of such an algorithm is shown in FIG. 26, which is a logical flow diagram of a process 3100 for generating a fused image using a plurality of image frames. In the following description of process 3100, reference should also be made to FIGS. 2 and 27-29. Process 3100, when executed by control circuit 132, can be embodied, for example, as instructions stored in a memory 134 coupled to control circuit 132 that cause control circuit 132 to execute the enumerated steps of process 3100. For the sake of brevity, process 3100 is described as being executed by control circuit 132, but it should be understood that process 3100 can be executed by other combinations of hardware, software, and / or firmware.

[0165] Therefore, as described above, the control circuit 132 that executes process 3100 can generate (3102) an image of the surgical site (e.g., via imaging system 142), and then determine (3104) whether the image is at least partially occluded. For example, FIG. 27 is a diagram of a series 3150 of n image frames 3160 captured by imaging system 142. The nth image frame 3160 can be the last captured image frame 3160, the (n - 1)th image frame 3160 can be the image frame 3160 captured immediately before it, and so on. Each of the image frames 3160 includes some pixels 3151 that may or may not correspond to pixels or cells of image sensor 135, for example. As seen in FIG. 27, the image frame 3160 can include an unoccluded portion 3162 and an occluded portion 3164. Specifically, when evaluating the nth image frame 3160, the control circuit 123 that executes process 3100 determines that the nth image frame 3160 is at least partially occluded because it includes the occluded portion 3164 of pixels 3151.

[0166] If the control circuit 132 determines 3104 that the image is not at least partially occluded, process 3100 proceeds along the no branch, and the control circuit 132 can continue to cause the imaging system 142 to generate an image (i.e., visualize the surgical site) for visualization of the surgical site, as described above. If the control circuit 132 determines 3104 that the image is at least partially constituted (e.g., as shown in the nth image frame 3160), the process proceeds along the yes branch, and the control circuit 132 can retrieve 3106 a previous image from the image set 3150. In one aspect, the control circuit 132 can continuously retrieve 3106 one or more previous images from the image set 3150 until the control circuit 132 places a corresponding unoccluded image portion that replaces the occluded portion of the first image.

[0167] Accordingly, the control circuit 132 can generate an updated image (3108) from the original image and one or more previous images retrieved from the image set 3150. For example, FIGS. 28 and 29 show an updated or fused image 3152 generated from a plurality of consecutive image frames 3160. In this particular example, n is equal to 60, but this is for illustrative purposes only. In FIG. 28, the numbers shown within each pixel 3151 correspond to the image frame 3160 from which the particular pixel 3151 was extracted. As can be seen, the fused image 3152 is generated from a combination of pixels 3151 across several different image frames 3160. Specifically, as shown in FIG. 27, image frames 3160 55 to 60 respectively correspond to the (n - 5)th to nth image frames 3160 in order. Accordingly, the control circuit 132 is configured to repeatedly retrieve previous images from the image set 3150 taken in by the imaging system 142, as in 3106, and to extract image portions such as pixels 3151 corresponding to pixels 3151 that are not blocked in the retrieved image but are blocked in consecutive images. The control circuit 132 can repeat this process until a set of image portions that are completely or substantially unblocked from the image set 3150 are retrieved and then the image portions are fused together to generate an updated image, as in 3108. The resulting fused image 3152 generated using this technique is shown in FIG. 29, demonstrating how structures 3140 such as tumors 3038 and blood vessels are made visible to the user from the initially partially blocked images.

[0168] Another technique that can be utilized to compensate for obscurations present at the surgical site is to perform a comparative analysis of a set of synchronized imaging devices. The control system 133 can be configured to interlace a plurality of image portions generated by a plurality of synchronized imaging devices to generate a fused image. In particular, a portion of an image generated by a first or primary imaging system (e.g., the imaging system 142 shown in FIG. 2) can be replaced with a corresponding portion of an image generated by a secondary imaging system. In particular, a set of imaging systems can be configured to time-index their scans. Obscured, damaged, unclear, or otherwise interfered-with portions of a first scan generated by a first imaging system can be replaced with clearer and / or verified portions of a second scan (time-indexed according to the first scan) generated by a second imaging system. If image data is missing, damaged, or obscured in the imaging of a primary dynamic data set generated by a first imaging system, a secondary scan from another imaging system (which can also sense at a different wavelength or wavelength range) can be utilized by the control system 133 to sharpen, replace, or interpolate the primary image to improve visualization of the surgical site for the user.

[0169] Surgical System Control Based on Multiple Sensed Parameters One challenge common to any surgical procedure and surgical instrument is to control the surgical instrument in an optimal manner for a given patient and / or tissue state. To that end, some surgical instruments include sensors for sensing various parameters associated with the surgical instrument and / or the tissue being operated on by the surgical instrument. However, some sensed data may indicate different states or situations of the tissue and thus may be inconclusive in the absence of additional data. Accordingly, a surgical system can incorporate data from an imaging system with other sensed data to resolve ambiguities and optimally control the surgical instrument according to the determined state / situation of the tissue.

[0170] In one general aspect, the present disclosure relates to a control system configured to utilize two sources of related but not identical data sources to distinguish different states of tissue being acted upon by a surgical instrument. Such states include, for example, fluid flow within the tissue and the thermal effects of energy directed by the surgical instrument onto the tissue. The control system can be configured to control a surgical instrument such as the surgical instrument 3290 described below.

[0171] FIG. 30 is a schematic diagram of a surgical instrument 3290 configured to control various functions, according to at least one aspect of the present disclosure. In one aspect, the surgical instrument 3290 is programmed to control the distal translation of a displacement member such as a closure member 3264. The surgical instrument 3290 includes an end effector 3292 that can include a clamp arm 3266, a closure member 3264, and an ultrasonic blade 3268 that can be exchanged with or operate in conjunction with one or more RF electrodes 3296 (shown in dashed lines). The ultrasonic blade 3268 is coupled to an ultrasonic transducer 3269 driven by an ultrasonic generator 3271.

[0172] In one aspect, the sensor 3288 may be implemented, among other things, as a limit switch, an electromechanical device, a solid state switch, a Hall effect device, an MR device, a GMR device, a magnetometer. In other implementations, the sensor 3288 may be, among other things, a solid state switch that operates under the influence of light such as an optical sensor, an IR sensor, an ultraviolet sensor. Further, the switch may be a solid state device such as a transistor (e.g., FET, junction FET, MOSFET, bipolar, etc.). In other implementations, the sensor 3288 may include, among other things, a non-electric conductor containing switch, an ultrasonic switch, an accelerometer, and an inertial sensor.

[0173] In one aspect, the position sensor 3284 may be implemented as an absolute positioning system including a magnetic rotary absolute positioning system implemented as the AS5055EQFT single-chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 3284 may be interfaced with the control circuit 3260 to provide an absolute positioning system. The position is located above the magnet and is provided to implement a concise and efficient algorithm for calculating hyperbolic and trigonometric functions that requires only addition, subtraction, bit shifting, and table reference operations, and may include a plurality of Hall effect elements coupled to a CORDIC processor, also known as the digit-by-digit method and the border algorithm.

[0174] In some examples, the position sensor 3284 may be omitted. If the motor 3254 is a stepper motor, the control circuit 3260 can track the position of the closure member 3264 by summing the number and direction of steps the motor has been instructed to execute. The position sensor 3284 may be located within the end effector 3292 or at any other part of the instrument.

[0175] The control circuit 3260 can communicate with one or more sensors 3288. The sensors 3288 are positioned on the end effector 3292 and may be adapted to operate with the surgical instrument 3290 to measure various derived parameters such as gap distance versus time, tissue compression versus time, and anvil strain versus time. The sensors 3288 may include inductive sensors such as magnetic sensors, magnetic field sensors, strain gauges, pressure sensors, force sensors, eddy current sensors, resistance sensors, capacitive sensors, optical sensors, and / or any other suitable sensors for measuring one or more parameters of the end effector 3292. The sensors 3288 may include one or more sensors.

[0176] The RF energy source 3294 is coupled to the end effector 3292 and is applied to the RF electrode 3296 when the RF electrode 3296 is provided within the end effector 3292 instead of the ultrasonic blade 3268, or when provided to operate in conjunction with the ultrasonic blade 3268. For example, the ultrasonic blade may be made of a conductive metal and may be used as a return path for an electrosurgical RF current. The control circuit 3260 controls the delivery of RF energy to the RF electrode 3296.

[0177] Additional details are disclosed in U.S. Patent Application No. 15 / 636,096, filed June 28, 2017, entitled "SURGICAL SYSTEM COUPLABLE WITH STAPLE CARTRIDGE AND RADIO FREQUENCY CARTRIDGE, AND METHOD OF USING SAME", which is hereby incorporated by reference in its entirety.

[0178] In various aspects, the sensor 3288 of the surgical instrument 3290 can include sensors configured to detect or measure various electrical parameters associated with tissue acted upon by the surgical instrument 3290, such as the capacitance or impedance of the tissue. In various aspects, the sensor 3288 can also include sensors configured to detect or measure various physical parameters associated with tissue acted upon by the surgical instrument 3290, such as temperature, viscoelastic compression (e.g., tissue creep, time to stabilization, or rate of initial load), or thickness (e.g., detectable upon first contact of the jaw with the tissue). Further, the image sensor 135 of the control system 133 shown in FIG. 2 can be utilized to detect or measure various tissue parameters based on the EMR emitted by the imaging system 142 using the various techniques described above. For example, the image sensor 135 can be configured to detect the refractive index of tissue at various wavelengths, the polarization of EMR / light reflected by the tissue, the passive IR radiation by the tissue, or the Doppler wavelength shift associated with the tissue. Any of these imaging-based parameters can be utilized in conjunction with other sensed parameters (e.g., electrical or physical parameters) to confirm the state or condition of tissue that cannot be directly confirmed individually via the parameter.

[0179] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system based on the state or condition of the tissue being acted upon by the surgical instrument, where the parameters can be determined based on parameters sensed by the imaging system and other sensors. An example of such an algorithm is shown in FIG. 31, which is a logical flow diagram of a process 3300 for controlling the surgical system according to the sensed parameters. In the following description of process 3300, reference is also made to FIGS. 2 and 30. Process 3300, when executed by control circuit 132, can be embodied as instructions, for example, stored in a memory 134 coupled to control circuit 132 that causes control circuit 132 to execute the recited steps of process 3300. For simplicity, process 3300 is described as being executed by control circuit 132, but it should be understood that process 3300 can be executed by other combinations of hardware, software, and / or firmware.

[0180] Accordingly, control circuit 132 can receive 3302 a measurement of a first tissue parameter via imaging system 142. As described above, the first tissue parameter can include, for example, the refractive index of the tissue at various wavelengths, the polarization of light reflected by the tissue, the passive IR radiation by the tissue, or the Doppler wavelength shift associated with the tissue.

[0181] Accordingly, control circuit 132 can receive 3304 a measurement of a second tissue parameter via sensor 788. As described above, the second tissue parameter can include, for example, various electrical and / or physical properties of the tissue such as the temperature of the tissue, the viscoelastic compression, or the thickness.

[0182] Accordingly, the control circuit 132 can determine (3306) the state or condition of the tissue based on the combination of measured values of the received tissue parameters 3302, 3304, and then control (3308) the surgical instrument 3290 accordingly. The same measured values of various electrical and / or physical properties of the tissue may indicate different states of the tissue, which may in turn require different control adjustments to be applied to the surgical instrument 3290. In the absence of additional or supplementary information, the control system 133 may not be able to correctly control the surgical instrument 3290 for a given state of the tissue in situations where the tissue state is ambiguous based on the measured values of the various electrical and / or physical properties of the tissue. Accordingly, the control system described herein supplements the electrical and / or physical properties sensed by the sensor 3288 with tissue parameters sensed via the imaging system 142 in order to accurately identify the state or condition of the tissue and then appropriately control the surgical instrument 3290. For example, in response to detecting an increase in the tissue temperature (i.e., the second tissue parameter 3304 received during process 3300) localized at the end effector 3292 of the surgical instrument 3290, various methods of controlling the surgical instrument 3290 may be appropriate. If the control system 133 detects a corresponding change in the polarization or refractive index of the tissue (i.e., the first tissue parameter 3302 received during process 3300), the control circuit 132 can determine (3306) that the tissue is undergoing associated thermal damage and control (3308) the surgical instrument 3290 to reduce the instrument power level or provide a proposal to the user to reduce the instrument power level. Conversely, if no corresponding change in the polarization or refractive index of the tissue is detected, the control circuit 132 can determine (3306) that the tissue is not undergoing associated thermal damage and control (3308) the surgical instrument 3290 to maintain or increase the instrument power level or provide a proposal for this effect. As another example, in response to detecting the tissue impedance (i.e., the second tissue parameter 3304 received during process 3300) of the tissue grasped by the end effector 3292 of the surgical instrument 3290, various methods of controlling the surgical instrument 3290 may be appropriate.While the imaging system 142 visualizes tissue movement, creep, or compression (i.e., the first tissue parameter 3302 received during process 3300), if the control system 133 does not detect a change in tissue impedance, the control circuit 132 can determine (3306) that there is a subsurface irregularity in the grasped tissue.

[0183] The control system 133 described herein can be implemented or executed, for example, in a surgical instrument 3290, a surgical hub 2236 (FIG. 21) to which the surgical instrument 3290 can be communicatively coupled (e.g., an energy device 3241 as shown in FIG. 21), or a combination thereof (e.g., using a distributed processing protocol). When the control system is embodied as a component of the surgical instrument 3290, the imaging data can be received directly from the imaging system 142 or via either the surgical hubs 2106, 2236 (FIGS. 17 - 19) next coupled to the imaging system 142. When the control system 133 is embodied as a component of the surgical hubs 2106, 2236, the imaging data can be received from the imaging system 142 coupled to the surgical hubs 2106, 2236, the surgical instrument sensor data can be received from the surgical instrument 3290 coupled to the surgical hubs 2106, 2236, and then the control system 133 of the surgical hubs 2106, 2236 can determine and transmit to the surgical instrument 3290 for execution appropriate surgical instrument control adjustments.

[0184] Adaptive optics for compensating imaging artifacts In one aspect, a control system, such as control system 133 described in connection with FIG. 2, can be configured to compensate for imaging artifacts associated with an imaging system 142 coupled thereto. In one aspect, control system 133 can be configured to adjust the optical signals received by imaging system 142 over a plurality of light wavelengths, in combination with selective imaging segment selection within a sample rate exceeding 60 Hz, to remove optical particulate obstructions from visualization. In one aspect, control system 133 can be configured to emit a control beam projected (e.g., via imaging system 142) and, correspondingly, monitor return signals on separate frames of a scanning array (e.g., of image sensor 135) to determine EMR / light distortion by particulates within a gas occupying a body cavity. The dispersion of the control projection from its projection source gives control system 133 a baseline for adjusting the scope visualization frame in a later part of the scan.

[0185] Surgical System Control Based on Suspended Particle Characteristics One problem inherent in surgical procedures using electrosurgical instruments is the smoke generated by the instruments. Surgical smoke can contain toxic gases and vapors, dead and live cell materials, blood fractions, and bioaerosols including viruses, as well as mutagenic and carcinogenic compounds. Therefore, it is highly desirable to remove these particulates from the surgical site, and thus, smoke evacuators are generally utilized in surgical procedures that result in the generation of surgical smoke. However, different particulate types may require different types of control adjustments to accurately control and reduce smoke generation during a surgical procedure, and thus, it is desirable to control the smoke evacuator and other surgical devices (including surgical instruments) depending on the type of particulates being generated. The surgical system can, for example, change the surgical instrument energy profile to result in less smoke generation and / or automatically control the smoke evacuator according to the type of particulates being generated.

[0186] In one general aspect, the present disclosure relates to a control system configured to detect the level of polarization of light emitted by an imaging system to determine parameters of a particulate cloud and to adjust control parameters of an associated system or device accordingly. In a further aspect, the polarization of the EMR reflected from the detected particulates can be utilized in combination with the vectorization and amount of the generated particulates to determine the source of the particulates, which can then be utilized to control a device that causes the generation of particulates to improve visualization at the surgical site. In yet a further aspect, it can be determined whether it is more effective to utilize the polarization of the EMR reflected from the detected particulates to adjust control parameters of an electrosurgical instrument or a smoke evacuator when improving visualization at the surgical site.

[0187] FIG. 32 is a diagram of a polarized EMR source 3500 for detecting different particulate types, according to at least one aspect of the present disclosure. The polarized EMR source 3500 can include an emitter 3502 configured to emit EMR 3506 and a polarization filter 3504 configured to polarize the emitted EMR 3506. The polarization filter 3504 can be removably fixable or integrally attachable to the emitter 3502. The polarized EMR source 3500 can be embodied as a component of an imaging system that can include, for example, the surgical visualization system 100 shown in FIG. 1, the imaging system 142 shown in FIG. 2, and / or the surgical visualization system 500 shown in FIG. 5. Accordingly, the emitter 3502 can include, for example, the emitter 106 shown in FIG. 1, the structured light source 152 shown in FIG. 2, the spectral light source 150 shown in FIG. 2, and the like. Next, the imaging system can be embodied as a component of a surgical system, such as the robotic surgical system 110 shown in FIG. 1, that can further include a control system configured to control various aspects of the surgical system. Examples of the control system can include the control system 133 shown in FIG. 2 and / or the control system 600 shown in FIG. 11.

[0188] During a surgical procedure, airborne particles may be present at the surgical site. These particles can include both naturally occurring particles and non-natural or synthetic particles. Naturally occurring particles can be generated by the interaction between a surgical instrument, such as an electrosurgical instrument, and the tissue being treated. Naturally occurring particles can include, for example, dead and living cell material, blood fractions, and other biological materials. Artificial or synthetic particles can be introduced into the surgical site by surgical staff. These particles can be embodied as smoke or aerosol present within or at the surgical site. Generally speaking, the presence of such particles may be undesirable, and thus many surgical systems include a smoke evacuator to remove undesirable smoke or aerosol from the surgical site. However, an imaging system can be configured to detect particles (i.e., smoke) generated at the surgical site, and a control system can be configured to control various operating parameters of the surgical system or its components based on the characteristics or properties of the detected particles. Some examples of such control algorithms are described herein.

[0189] Referring back to FIG. 32, as described above, the airborne particulates at the surgical site can include both naturally occurring particulates 3510 and artificial particulates 3512. Since different operations may be required to reduce the presence of each of the different types of particulates, it may be beneficial to be able to distinguish between the different types of particulates present at the surgical site. For example, if the type of particulate detected is the naturally occurring particulate 3510 that can be generated from an electrosurgical instrument treating tissue, it may be desirable to control the electrosurgical instrument to reduce the generation of the naturally occurring particulate 3510 (e.g., by reducing the energy duty cycle or otherwise changing the energy delivery profile of the instrument). Conversely, if the detected particulate type is the synthetic particulate 3512, since that particulate type is not generated by the operation of the electrosurgical instrument, controlling the electrosurgical instrument will not affect the presence of the synthetic particulate 3512. Instead, it may be desirable to increase the suction flow rate of the smoke evacuator to remove the synthetic particulate 3512 from the surgical site. Further, if a combination of different particulate types is detected at the surgical site, it may be desirable to control the electrosurgical instrument and the smoke evacuator in combination with each other and vary the control adjustments for each device. Thus, the control system of the surgical system can be configured to detect the different types of airborne particulates present at the surgical site and to control the various devices or components of the surgical system to reduce or eliminate the particulates from the surgical site.

[0190] In one aspect, the naturally occurring particles 3510 and the synthetic particles 3512 can be distinguished from each other based on the reflection characteristics of the suspended particles 3510, 3512 when exposed to polarized EMR 3506. For example, the control system 133 determines the distance to the body undergoing the surgical procedure and defines a range gate for the emitter 3502 to obtain depolarization measurements only from the EMR 3506 reflected from particles within the air space between the emitter 3502 and the body cavity rather than from within the body cavity itself. To do so, the emitter 3502 can be configured to pulse coherent EMR at a plurality of different wavelengths regardless of the presence or absence of polarization. In particular, the control system 133 can be configured to pulse coherent EMR at a first wavelength and a second wavelength at the emitter 3502. The first wavelength can be selected such that the EMR at the first wavelength is substantially non-interactive with the naturally occurring particles 3510 and the synthetic particles 3512 and can thus pass through the smoke and be reflected from the body cavity. The control system 133 can then determine the distance to the body cavity via the time-of-flight sensor systems 1104, 1204 according to, for example, the time difference between when the EMR is emitted and when the reflected EMR is detected, as described in connection with FIGS. 14-16. The second wavelength can be selected such that the EMR at the second wavelength is substantially interactive with the naturally occurring particles 3510 and the synthetic particles 3512 and can thus be utilized to detect or measure characteristics associated with different particle types. Thus, the control system 133 can use the body cavity distance determined by pulsing EMR at the first wavelength to range gate the measurements received by the EMR at the second wavelength to ensure that only measurements of the suspended particles are being made. The control system 133 can then determine whether the suspended particles are the naturally occurring particles 3510 and / or the synthetic particles 3512 based on the reflection characteristics of the suspended particles 3510, 3512 and can control other components of the surgical system accordingly.

[0191] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system based on the type of floating particles detected at the surgical site. An example of such an algorithm is shown in FIG. 33A, which is a logical flow diagram of a process 3600 for controlling the surgical system according to the detected particle type, and FIG. 33B is a logical flow diagram of a process 3650 for controlling the surgical system according to the detected particle type detected within a defined range gate. In the following description of processes 3600 and 3650, also refer to FIG. 2. Processes 3600 and 3650, when executed by control circuit 132, can be embodied as instructions, for example, stored in a memory 134 coupled to control circuit 132 that cause control circuit 132 to execute the enumerated steps of processes 3600 and 3650. For simplicity, processes 3600 and 3650 are described as being executed by control circuit 132, but it should be understood that processes 3600 and 3650 can be executed by other combinations of hardware, software, and / or firmware.

[0192] Referring now to FIG. 33A, control circuit 132 executing process 3600 can cause imaging system 142 to emit 3602 polarized EMR directed towards the surgical site, for example, via polarized EMR source 3500.

[0193] Accordingly, the control circuit 132 can receive 3604 the polarized EMR reflected from the floating microparticles at the surgical site (e.g., within the body cavity) and determine 3606 whether the detected microparticle type is a naturally occurring microparticle 3510 or an artificial microparticle 3512. The control circuit 132 can distinguish different types of floating microparticles 3510, 3512 based on their different reflection characteristics when exposed to polarized EMR. In particular, one of the types of floating microparticles 3510, 3512 (e.g., artificial microparticles 3512) can scatter polarized EMR at a higher rate than the other (e.g., naturally occurring microparticles 3510). This can reduce the degree of visualization of the scattered floating microparticle type or otherwise affect the manner in which the reflected EMR is received by the image sensor 135 of the imaging system 142. Accordingly, this difference in the visualization of different types of floating microparticles 3510, 3512 can be characterized and utilized to identify the types of floating microparticles 3510, 3512 present at the surgical site (e.g., within the body cavity).

[0194] Accordingly, if the microparticles are naturally occurring microparticles 3510, the process 3600 can proceed along the yes branch, and the control circuit 132 can adjust 3608 the control parameters of the surgical system to a first state corresponding to the naturally occurring microparticles 3510. Conversely, if the microparticles are artificial microparticles 3512, the process 3600 can proceed along the no branch, and the control circuit 132 can adjust 3610 the control parameters of the surgical system to a second state corresponding to the artificial microparticles 3512.

[0195] In another aspect, various combinations of naturally occurring microparticles 3510 and artificial microparticles 3512 can exist within the body cavity. In such cases, the control circuit 132 can instead determine the relative ratio of the types of floating microparticles 3510, 3512 present within the body cavity (e.g., due to the relative degree to which visualization is reduced or affected) and then control the surgical device or combination of surgical devices accordingly.

[0196] In yet another aspect, the control system may be configured to range gate measurements and / or visualizations based on the polarized EMR source 3500. Such an aspect can be implemented by the process 3650 shown in FIG. 33B.

[0197] Accordingly, the control circuit 132 that executes the process 3650 can cause the imaging system 142 to emit EMR at a first wavelength directed at the surgical site, for example, via the structured light source 152 and / or the spectral light source 150. In one aspect, the first wavelength can be a wavelength that is substantially non-interactive with the naturally occurring microparticles 3510 and the synthetic microparticles 3512 imaged by the imaging system 142.

[0198] Accordingly, the control circuit 132 receives 3654 the EMR reflected from the body cavity (i.e., the surgical site) via the imaging system 142 and defines 3656 a range gate corresponding to the space between the emitter of the imaging system 142 and the body cavity surface as described above.

[0199] Accordingly, the control circuit 132 can cause the imaging system 142 to emit polarized EMR at a second wavelength directed at the surgical site, for example, via the polarized EMR source 3500. In one aspect, the second wavelength can be a wavelength that is substantially interactive with the naturally occurring microparticles 3510 and the synthetic microparticles 3512 imaged by the imaging system 142.

[0200] Accordingly, the control circuit 132 can receive the polarized EMR reflected within the defined range gate that can correspond to the suspended microparticles 3510, 3512 located between the emitter of the imaging system 142 and the body cavity surface via the imaging system 142. The control circuit 132 can then determine 3662 whether the detected particle type is the naturally occurring microparticle 3510 or the artificial particle 3512 and adjust 3608, 3610 the control parameters of the surgical system to a first state or a second state as described above with respect to the process 3600 shown in FIG. 33A.

[0201] In another aspect, the surgical system can be further configured to track the movement of the suspended particles throughout the surgical procedure, which can then be used to characterize the movement and size or composition changes of the cloud defined by the suspended particles. By characterizing the movement of the suspended particles over time, the surgical system can, for example, determine how well smoke evacuation from the body cavity is being performed and then adjust or provide recommendations to the user to adjust the location or magnitude of the smoke evacuation or air delivery. For example, the surgical system can stop the first smoke evacuator or the first air delivery device and activate the second smoke evacuator or the second air delivery device to adjust the gas circulation current within the body cavity, thereby alleviating any vortices within the body cavity (i.e., regions where the movement vectors of the particles and / or gas are zero or near zero) and improving the smoke evacuation performance. As another example, the surgical system can adjust the motor level or fan level of the smoke evacuator or inhaler to improve the smoke evacuation performance.

[0202] In one aspect, the control system 133 can be configured to track and characterize the movement of the suspended particles by characterizing the detection of the particles across the cells or pixels of the image sensor 135. In particular, the control system 133 can determine which pixels of the image sensor 135 detected the particles and then track the movement of the particles over time across the pixel array of the image sensor 135. In one aspect, the control system 133 can be configured to divide the image obtained via the image sensor 135 into two or more pixel array sections, generate a motion vector corresponding to the generalized change in position due to the detected suspended particles from a first time instance to a second time instance, and then characterize the movement or composition change of the particle cloud accordingly.

[0203] For example, FIGS. 34A-34C show a pixel array 3700 of an image sensor 135 consisting of several pixels 3701. Further, FIGS. 34A-C show the change over time of the detected particle positions and the generalized particle cloud motion vectors calculated therefrom. The pixel array 3700 is depicted as a 5×5 array, but this is for illustrative purposes only, and it should be noted that neither the image sensor 135 nor the selected sub-section of its pixels is limited to a 5×5 array. In the following description of FIGS. 34A-34C, reference is also made to FIG. 2.

[0204] FIG. 34A shows the detection array 3700 at time t1, which shows the detection of a first particle 3702a, a second particle 3702b, a third particle 3702c, a fourth particle 3702d, and a fifth particle 3702e in the shown pixels 3701 of the image sensor 135. FIG. 34B shows the detection array 3700 at time t2, which shows that the first and fifth particles 3702a, 3702e did not change position, while the second, third, and fourth particles 3702b, 3702c, 3702d were detected in different pixels 3701 of the image sensor 135. Based on the detected movement of the particles, the control circuit 132 can be configured to determine a vector representation of the movement of each of the pixels. The vector representation can include both direction and magnitude. Based on the direction and magnitude of the movement vectors, the control circuit 132 can be further configured to calculate (e.g., using vector addition) a vector 3704 corresponding to the generalized movement of the cloud defined by the detected particles. Thus, the control circuit 132 can track the change of the particle cloud or aerosol from a first state 3710 (e.g., a first position or a first size) to a second state 3712 (e.g., a second position or a second size) as shown in FIG. 35, according to the vector 3704 calculated from the change of the pixels of the image sensor 135 where the particles are detected.

[0205] In one aspect, a control circuit 132 coupled to the image sensor 135 is configured to calculate a generalized motion vector corresponding to a change in position of detected airborne particles within a pixel array 3700 (which can represent the entire pixel array of the image sensor 135 or a sub-section thereof) so as to track the movement of the detected airborne particles, and then to accordingly control various connected surgical devices such as an air injector, a smoke evacuator, and / or a surgical instrument. In one aspect, a control system 133 including the control circuit 132 can be embodied as surgical hubs 2106, 2236 as described above in the context of the surgical hub system. In this aspect, the surgical devices are communicatively connected (e.g., paired) to the surgical hubs 2106, 2236 and can be controlled according to the described systems and processes.

[0206] In another aspect, the control system 133 can be configured to utilize Raman spectroscopy techniques to determine the vibrational / rotational modes of the suspended microparticles, for example, using near-infrared, UV, or a combination of near-infrared and UV wavelengths. Data obtained from such techniques can provide information regarding, for example, gas-phase species (e.g., benzene vs. aldehyde), which in turn can provide insights regarding the type of tissue from which the particles were generated or the efficiency of the energy applied to the tissue. Since the desired information about the species is contained in the inelastic scattering of the EMR, the control system 133 can include, for example, a filter (e.g., a bandpass or notch filter) coupled to the detector to filter the elastic scattering of the source EMR. Signals generated by the image sensor 135 or another such detector (e.g., a CCD detector) according to Raman spectroscopy techniques can be based on the unique structural characteristics of the detected molecules. In particular, Raman spectroscopy is based on the concept that, for example, photons emitted by a suitable emitter excite a molecule to a higher energy state, such that the scattered photons change frequency as a result of conserving energy from the vibrational / rotational changes of the molecule. This change in the frequency of the scattered photons can be utilized to characterize the type of molecule with which the photons interacted by comparing the detected signal with pre-characterized data of a given excitation frequency according to the particular type of monochromatic light source utilized. The determined molecular type of the microparticles can be utilized for several different applications, including providing specific data regarding the relative amounts of potentially dangerous molecules generated at the surgical site for safety monitoring purposes. The determined molecular type of the microparticles can also be utilized to evaluate the effectiveness and integrity of a smoke evacuation system or its filter.

[0207] Surgical System Control Based on Smoke Plume Characteristics One problem unique to surgical procedures using electrosurgical instruments is the smoke generated by the instruments. Surgical smoke can contain toxic gases and vapors, dead and live cell materials, blood fractions, and bioaerosols including viruses, as well as mutagenic and carcinogenic compounds. Thus, it is highly desirable to remove these particulates from the surgical site, and thus, smoke evacuators are generally utilized in surgical procedures that result in the generation of surgical smoke. However, in order to accurately control and reduce the generation of smoke during a surgical procedure, it is desirable to control the smoke evacuator and other surgical devices (including surgical instruments) according to the amount of smoke at the surgical site, the variation of the smoke cloud over time (e.g., whether the smoke cloud is actively accumulating or decreasing), and other such smoke cloud characteristics. The surgical system can, for example, change the surgical instrument energy profile to generate less smoke and / or automatically control the smoke evacuator according to the amount of surgical smoke generated.

[0208] In one general aspect, the present disclosure is directed to a surgical system configured to detect and characterize an amorphous three-dimensional particulate cloud generated during a surgical procedure. The surgical system can be configured to detect the movement of the particulate cloud with respect to the intraperitoneal cavity and the surgical site and then, in response, control various surgical devices such as surgical instruments or smoke evacuators. In one general aspect, the present disclosure is directed to a control system configured to define the surface or boundary of a cloud or particulate cluster generated during a surgical procedure and to analyze various characteristics of the defined cloud, such as the direction and rate of change of the boundary, to control various control parameters of the surgical system, such as the power level of the surgical instrument / generator or the smoke evacuation motor control. In a further aspect, the control system can be configured to generate a boundary by defining a predefined density of the particulates based on the overall volume of the particulates or the size of the particulates. In another further aspect, the rate of change of the particulate cloud surface boundary can be utilized to directionally define the rate of change of the energy device or the smoke evacuation mechanism.

[0209] FIG. 36 is a diagram of a surgical system 3750 during a surgical procedure in which a particulate cloud 3752 is being generated, according to at least one aspect of the present disclosure. The surgical system 3750 can be embodied as a robotic surgical system, such as the robotic surgical system 110 shown in FIG. 1, for example. The surgical system 3750 can include an electrosurgical instrument 3754, a smoke evacuator 3756, a grasper 3750, and any other surgical device for treating, cutting, or otherwise manipulating tissue 3760 for a surgical procedure. Although not shown in FIG. 36, the surgical system 3750 can further include an imaging system that can include, for example, the surgical visualization system 100 shown in FIG. 1, the imaging system 142 shown in FIG. 2, and / or the surgical visualization system 500 shown in FIG. 5. The surgical system 3750 can further include a control system that can include, for example, the control system 133 shown in FIG. 2 and / or the control system 600 shown in FIG. 11.

[0210] During a surgical procedure, airborne particulates 3751 can be generated by the interaction between a surgical instrument, such as the electrosurgical instrument 3754, and the tissue 3760 being treated. These particulates 3751 can be embodied as a cloud of smoke or aerosol present within or at the surgical site. Generally speaking, the presence of such particulates 3751 can be undesirable, and thus many surgical systems 3750 include a smoke evacuator 3756 for removing particulates 3751 from the surgical site. However, the imaging system can be configured to image the particulates 3751 and / or smoke generated at the surgical site, and the control system can be configured to control various operating parameters of the surgical system 3750 or its components based on the characteristics or qualities of the imaged smoke. Some examples of such control algorithms are described herein.

[0211] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system 3750 based on one or more characteristics associated with the plume generated at the surgical site. An example of such an algorithm is shown in FIG. 37, which is a logical flow diagram of a process 3800 for controlling the surgical system according to particulate cloud characteristics. Refer also to FIG. 2 in the following description of process 3800. Process 3800, when executed by control circuit 132, can be embodied, for example, as instructions stored in memory 134 coupled to control circuit 132 that cause control circuit 132 to execute the enumerated steps of process 3800. For brevity, process 3800 is described as being executed by control circuit 132, but it should be understood that process 3800 can be executed by other combinations of hardware, software, and / or firmware.

[0212] Accordingly, control circuit 132, which executes process 3800, can detect 3802 the presence of airborne particles within the FOV of imaging system 142 using any of the techniques described above. Generally, image sensor 135 of imaging system 142 can detect EMR emitted by structured light source 152 and / or spectral light source 150 and reflected by airborne particles to detect / image the particles.

[0213] Accordingly, control circuit 132 can characterize 3804 the particulate cloud defined by the detected particles. In one aspect, control circuit 132 can be configured to depict an amorphous three-dimensional construct that defines the three-dimensional boundary of the particulate cloud and can track its density, volume, position, movement, and / or boundary over time. The boundary of the particulate cloud at the surgical site can be defined in a variety of different ways. For example, the particulate cloud boundary can be defined as the volume encompassing all airborne particles detected within the FOV of imaging system 142. As another example, the particulate cloud boundary can be defined as the volume having a threshold density of airborne particles.

[0214] Accordingly, the control circuit 132 can determine 3806 whether one or more characteristics of the particulate cloud have breached a threshold. Such tracked characteristics can include, for example, the density of the particulate cloud, the volume of the particulate cloud, the position of the particulate cloud and / or its boundary, the movement of the particulate cloud and / or its boundary, and / or the rate of change or any other derivative of the aforementioned features. The threshold for the tracked characteristics may depend on other parameters such as the surgical situation (e.g., the type of surgical procedure being performed). If the threshold has not been breached, the process 3800 can proceed along the no branch, and the control circuit 132 can continue as described above, for example, until a stop criterion is met (e.g., the surgical procedure is complete). If the threshold has been breached, the process 3800 can proceed along the yes branch, and the control circuit 132 can continue as described below.

[0215] Accordingly, the control circuit 132 can adjust one or more control parameters of the surgical system 3750. Examples of control parameters that can be adjusted by the control circuit 132 include surgical instrument / generator energy levels, smoke evacuator suction, visualization parameters, and the like. For example, FIG. 38 shows a series of graphs 3850, 3852, 3854 depicting the adjustment of control parameters based on particulate cloud characteristics by the control circuit 132 executing process 3800. The first graph 3850 shows a first line 3860 depicting the change in smoke cloud density over time represented by the horizontal axis 3858 and the vertical axis 3856. The second graph 3852 shows a second line 3868 depicting the change in the energy duty cycle of the electrosurgical instrument 3754 (or the generator driving the electrosurgical instrument 3754) over time represented by the horizontal axis 3858 and the vertical axis 3866. The third graph 3854 shows a third line 3880 depicting the change in the smoke evacuation or suction flow rate of the smoke evacuator 3756 over time represented by the horizontal axis 3858 and the vertical axis 3878. Collectively, graphs 3850, 3852, 3854 show representative predictive implementations of process 3800 during a surgical procedure, where process 3800 adjusts the energy duty cycle of the electrosurgical instrument and the smoke evacuator suction flow rate control parameters in accordance with the characterized smoke cloud density.

[0216] Initially, as shown by the first graphic 3890, the electrosurgical instrument 3754 does not apply energy to the captured tissue 3760. Thus, the energy duty cycle of the electrosurgical instrument 3754 is zero, the smoke evacuator suction flow rate is at the base or default rate, and no smoke is generated (since no energy is being applied to the tissue 3760). At time t1, the surgeon activates the electrosurgical instrument 3754 and begins to apply energy to the tissue 3760 represented by an energy duty cycle that increases 3870 from 0 to E3. The application of energy to the tissue 3760 causes smoke to begin to be generated at the surgical site, which is represented by a smoke plume density that increases 3862 rapidly from 0 during the period after t1. Further, in response to the energy being activated, the smoke evacuator flow rate can be increased 3882 by the control circuit 132 from Q1 to Q2 when the smoke evacuator 3756 begins to attempt to remove the smoke generated from the surgical site. At this stage, the control circuit 132 can begin the detection 3802 of the particulates generated by the application of energy and the characterization 3804 of the corresponding smoke plume defined by the particulates.

[0217] At time t2, the application of energy to the tissue 3760 is generating a smoke plume 3752 at the surgical site as shown by the second graphic 3892. The control circuit 132 can determine 3806 that the cloud density has exceeded a smoke plume density threshold (e.g., represented by D3). Thus, the control circuit 132 adjusts 3808 by decreasing 3872 the electrosurgical instrument energy duty cycle control parameter from E3 to E2. The control circuit 132 can select this adjustment because applying a lower level of energy to the tissue 3760 can result in less smoke being generated. In response, the smoke plume density begins to decrease 3864 at time t1.

[0218] At time t3, as shown by the third graphic 3894, the smoke plume 3752 is decreasing in size but has not completely dissipated. The control circuit 132 can determine 3806 that the cloud density is not decreasing at a sufficient rate or that some other characteristic of the smoke plume is breaking some other threshold. Thus, the control circuit 132 readjusts 3808 by decreasing 3874 the electrosurgical instrument energy duty cycle control parameter from E2 to E1 to reduce further smoke generation.

[0219] At time t4, the smoke plume 3752 is nearly dissipated, as shown by the fourth graphic 3896. The control circuit 132 can determine 3806 that the smoke plume has broken another threshold, such as a cloud density above a particular level (represented, for example, by D1) for a period longer than a threshold period (represented, for example, by t4). Thus, the control circuit 132 adjusts by increasing 3884 the smoke evacuator suction flow rate control parameter from Q2 to Q3 to completely remove smoke particles from the surgical site.

[0220] Note that the implementation of the process 3800 embodied by FIG. 38 is provided for illustrative purposes and represents only one possible implementation. In particular, different control parameters can be controlled by the process 3800, different thresholds can be utilized, different smoke plume characteristics can be tracked, etc. Thus, FIG. 38 should in no way be construed as limiting the process 3800 of FIG. 37 or any of the other described systems and methods.

[0221] Exemplary Clinical Applications The various surgical visualization systems disclosed herein can be used in one or more of the following clinical applications. The following clinical applications are non-exhaustive and are merely exemplary uses for one or more of the various surgical visualization systems disclosed herein.

[0222] A surgical visualization system, as disclosed herein, can be used for many different types of procedures in different specialties such as, for example, urology, gynecology, oncology, colorectal, thoracic, bariatric / gastric surgery, and hepatopancreatobiliary (HPB). For example, in urological surgeries such as prostatectomy, the urinary tract may be detected in fat or connective tissue, and / or nerves may be detected, for example, in fat. For example, in gynecological oncology surgeries such as hysterectomy and colorectal surgeries such as low anterior resection (LAR), the ureter may be detected, for example, within fat and / or connective tissue. For example, in thoracic surgeries such as lobectomy, blood vessels may be detected in the lung or connective tissue, and / or nerves may be detected in connective tissue (e.g., esophageal fistula formation). In bariatric surgery, blood vessels may be detected in fat. For example, in HPB surgeries such as hepatectomy or pancreatomy, blood vessels may be detected in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts may be detected in parenchymal tissue (liver or pancreas).

[0223] In one example, a clinician may desire to remove uterine fibroids. From a preoperative magnetic resonance imaging (MRI) scan, the clinician can know that the uterine fibroids are located on the surface of the intestine. Thus, the clinician may desire to know during the surgery which tissues constitute a part of the intestine and which tissues constitute a part of the rectum. In such an example, a surgical visualization system, as disclosed herein, can show different types of tissues (intestine vs. rectum) and communicate that information to the clinician via an imaging system. Further, the imaging system can determine and communicate the proximity of a surgical device to the selected tissue. In such an example, the surgical visualization system can improve treatment efficiency without significant complications.

[0224] In another example, the clinician (e.g., a gynecologist) may remain away from a particular anatomical region to avoid getting too close to important structures, and thus the clinician may not remove, for example, all of the endometriosis. A surgical visualization system, as disclosed herein, can enable the gynecologist to reduce the risk of getting too close to important structures so that the surgical device can get close enough to remove all of the endometriosis and improve the patient's outcome (democratized surgery). Such a system can enable the surgeon to "keep moving" during the surgical procedure, rather than repeatedly stopping and resuming to identify areas to avoid, for example, during the application of therapeutic energy such as, in particular, ultrasonic or electro-surgical energy. In gynecological applications, the uterine arteries and ureters are important structures, and the system may be particularly useful for hysterectomy and endometrial surgery, taking into account the presentation and / or thickness of the tissues involved.

[0225] In another example, the clinician may risk incising a blood vessel in a location that could affect the blood supply to leaves other than the targeted leaf due to being too close. Further, anatomical differences between patients can result in the incising of blood vessels (e.g., branching vessels) that affect different leaves based on a particular patient. A surgical visualization system, as disclosed herein, can enable the identification of the correct blood vessel at the desired location, thereby enabling the clinician to reliably incise the appropriate anatomical object. For example, the system can confirm that the correct blood vessel is in the correct position, after which the clinician can safely divide the blood vessel.

[0226] In another example, due to the uncertain anatomical structure of the blood vessels, the clinician may make multiple incisions before making an incision at the best location. However, since more incisions may increase the risk of bleeding, it is desirable to make an incision at the best location in the first attempt. As disclosed herein, the surgical visualization system can minimize the number of incisions by indicating the correct blood vessels and the best location for the incision. For example, the ureter and the ligament are densely packed, presenting unique challenges during the incision. In such examples, it may be particularly desirable to minimize the number of incisions.

[0227] In another example, a clinician (e.g., an oncologic surgeon) who removes cancerous tissue may desire to know the identification of critical structures, the localization of the cancer, the cancer staging, and / or the assessment of tissue normality. Such information goes beyond what the clinician can see with the "naked eye". As disclosed herein, the surgical visualization system can determine and / or communicate such information to the clinician during the surgery, enhance intraoperative decision-making, and improve surgical outcomes. In certain examples, the surgical visualization system may be compatible with minimally invasive surgery (MIS), open surgery, and / or a robotic approach using either an endoscope or an exoscope, for example.

[0228] In another example, a clinician (e.g., an oncologic surgeon) may desire to turn off one or more warnings regarding the proximity of a surgical tool to one or more critical structures in order to avoid being overly conservative during the surgery. In other examples, the clinician may desire to receive a specific type of warning, such as tactile feedback (e.g., vibration / buzzer) indicating proximity and / or "no-fly zones", so as to remain sufficiently far away from one or more critical structures. As disclosed herein, the surgical visualization system can provide adaptability, for example, based on the clinician's experience and / or the desired aggressiveness of the procedure. In such examples, the system provides a balance between "knowing too much" and "knowing enough" to predict and avoid critical structures. The surgical visualization system can assist in planning the next steps during the surgery.

[0229] Various aspects of the subject matter described in this specification are illustrated in the following numbered examples.

[0230] Example 1. A control system for a surgical instrument, comprising: an imaging system; a sensor configured to detect a second tissue parameter of tissue acted upon by the surgical instrument; and a control circuit coupled to the imaging system and the sensor. The imaging system includes an emitter configured to emit electromagnetic radiation and an image sensor configured to receive the reflected electromagnetic radiation indicative of a first tissue parameter. The control circuit is configured to determine a state of the tissue based on a combination of the first tissue parameter and the second tissue parameter and to control the surgical instrument according to the determined state of the tissue.

[0231] Example 2. The control system of Example 1, wherein the first tissue parameter includes at least one of a tissue EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift.

[0232] Example 3. The control system of Example 1 or 2, wherein the second tissue parameter includes at least one of a tissue capacitance or impedance.

[0233] Example 4. The control system of Example 1 or 2, wherein the second tissue parameter includes at least one of a tissue temperature, viscoelastic compression, or thickness.

[0234] Example 5. The control system of any one of Examples 1 to 4, wherein the surgical instrument is selected from the group consisting of a surgical stapler, an electrosurgical instrument, and an ultrasonic surgical instrument.

[0235] Example 6. A control system for a surgical instrument, the surgical instrument including a sensor configured to detect a second parameter associated with the surgical instrument, the control system including an imaging system and a control circuit coupled to the imaging system and connectable to the sensor. The imaging system includes an emitter configured to emit electromagnetic radiation and an image sensor configured to receive the reflected electromagnetic radiation indicative of a first parameter of the tissue. The control circuit is configured to determine the state of the tissue based on a combination of the first parameter and the second parameter and to control the surgical instrument according to the determined state of the tissue.

[0236] Example 7. The control system according to Example 6, wherein the first parameter includes at least one of the EMR refractive index of the tissue, EMR polarization, passive IR radiation, or the Doppler wavelength shift of the tissue.

[0237] Example 8. The control system according to Example 6 or 7, wherein the second parameter includes at least one of the capacitance or impedance of the tissue.

[0238] Example 9. The control system according to Example 6 or 7, wherein the second parameter includes at least one of the temperature, viscoelastic compression, or thickness of the tissue.

[0239] Example 10. The control system according to any one of Examples 6 to 9, wherein the surgical instrument is selected from the group consisting of a surgical stapler, an electrosurgical instrument, and an ultrasonic surgical instrument.

[0240] Example 11. The control system according to any one of Examples 6, 7, or 10, wherein the second parameter is associated with a surgical device associated with the surgical instrument.

[0241] Example 12. The control system according to Example 11, wherein the surgical device includes a surgical generator to which the surgical instrument is connectable.

[0242] Example 13. The control system described in Example 12, wherein the second parameter includes the power level of the surgical generator.

[0243] Example 14. A surgical hub connectable to an imaging system, a surgical device, and a surgical instrument, the imaging system including an emitter configured to emit electromagnetic radiation at a surgical site and an image sensor configured to receive the electromagnetic radiation reflected from the surgical site, the reflected electromagnetic radiation indicating a first parameter associated with tissue, the surgical device including a sensor configured to detect a second parameter associated with the surgical device, the surgical hub configured to receive a first measurement of the first parameter from the imaging system, receive a second measurement of the second parameter from the sensor, determine a state of the tissue based on a combination of the first parameter and the second parameter, and control the surgical instrument according to the determined state of the tissue, the surgical hub including a control circuit.

[0244] Example 15. The surgical hub according to Example 14, wherein the first parameter includes at least one of the EMR refractive index of tissue, EMR polarization, passive IR radiation, or Doppler wavelength shift.

[0245] Example 16. The surgical hub according to Example 14 or 15, wherein the second parameter includes at least one of the capacitance or impedance of tissue.

[0246] Example 17. The surgical hub according to Example 14 or 15, wherein the second parameter includes at least one of the temperature, viscoelastic compression, or thickness of tissue.

[0247] Example 18. The surgical hub according to any one of Examples 14 to 17, wherein the surgical instrument is selected from the group consisting of a surgical stapler, an electrosurgical instrument, and an ultrasonic surgical instrument.

[0248] Example 19. The surgical hub according to any one of Examples 14, 15, or 18, wherein the surgical device includes a surgical generator to which the surgical instrument is connectable.

[0249] Example 20. The second parameter is the surgical hub described in Example 19, including the power level of the surgical generator.

[0250] Although several forms have been shown and described, it is not the intention of the applicant to limit or restrict the appended claims in such detail. Many modifications, variations, changes, substitutions, combinations, and equivalents of these forms can be implemented and would be envisioned by those skilled in the art without departing from the scope of the present disclosure. Further, the structure of each element related to the described forms can alternatively be described as a means for providing the function implemented by that element. Also, although materials are disclosed with respect to specific components, other materials may be used. Therefore, it should be understood that the foregoing description and the appended claims are intended to cover all such modifications, combinations, and variations as being within the scope of the disclosed forms. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications, and equivalents.

[0251] The above detailed description has described various forms of apparatus and / or processes using block diagrams, flow diagrams and / or examples. As long as such block diagrams, flow diagrams and / or examples include one or more functions and / or operations, it should be understood by those skilled in the art that each function and / or operation included in such block diagrams, flow diagrams and / or examples can be implemented individually and / or collectively by a variety of hardware, software, firmware or virtually any combination thereof. It should be understood by those skilled in the art that all or part of some of the forms disclosed herein can be implemented equivalently on an integrated circuit as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or in virtually any combination thereof, and that designing the circuitry and / or writing the code for the software and / or firmware is within the skill of those skilled in the art in view of the present disclosure. Additionally, it should be understood by those skilled in the art that the mechanisms of the subject matter described herein can be distributed in a variety of forms as one or more program products, and that the specific forms described herein apply regardless of the particular type of signal carrier medium used to actually carry out the distribution.

[0252] Instructions used to program logic to implement various disclosed aspects may be stored in system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Further, the instructions may be distributed via a network or by other computer-readable media. Thus, a machine-readable medium can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to floppy disks, optical disks, compact disks, read-only memory (CD-ROM), and magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic or optical cards, flash memory, or other tangible machine-readable storage used to transmit information via the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a non-transitory computer-readable medium can include any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0253] When used in any aspect of this specification, the term "control circuit" can refer to, for example, a hardwired circuit, a programmable circuit (e.g., a computer processor, processing unit, processor, microcontroller, microcontroller unit, controller, digital signal processor (DSP), programmable logic device (PLD), programmable logic array (PLA), or field programmable gate array (FPGA) that includes one or more individual instruction processing cores), a state machine circuit, firmware that stores instructions executed by a programmable circuit, and any combination thereof. The control circuit can be embodied, collectively or individually, as a circuit that forms part of a larger system, such as, for example, an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Thus, as used herein, "control circuit" refers to an electrical circuit having at least one individual electrical circuit, an electrical circuit having at least one integrated circuit, an electrical circuit having at least one application specific integrated circuit, an electrical circuit forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program that at least partially executes the processes and / or apparatuses described herein, or a microprocessor configured by a computer program that at least partially executes the processes and / or apparatuses described herein), an electrical circuit forming a memory device (e.g., in the form of a random access memory) and / or an electrical circuit forming a communication device (e.g., a modem, a communication switch, or an optical-electrical facility), but is not limited thereto. One of ordinary skill in the art will recognize that the subject matter described herein may be implemented in analog form, digital form, or some combination thereof.

[0254] As used in any aspect of this specification, the term "logic" may refer to an application, software, firmware, and / or circuitry configured to perform any of the foregoing operations. The software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. The firmware may be embodied as code, instructions, or instruction sets within a memory device, and / or hard-coded (e.g., non-volatile) data.

[0255] As used in any aspect of this specification, terms such as "component", "system", "module", etc. can refer to a computer-related entity that is either hardware, a combination of hardware and software, software, or software in execution.

[0256] As used in any aspect of this specification, an "algorithm" refers to a self-collision-free sequence of steps leading to a desired result, and a "step" refers to an operation of a physical quantity and / or logical state that, although not necessarily required, can take the form of an electrical or magnetic signal capable of being stored, transferred, combined, compared, and otherwise manipulated. It is common practice to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc. These and similar terms may be associated with appropriate physical quantities and are merely convenient labels applied to these quantities and / or states.

[0257] Examples of networks include packet-switching networks. Communication devices can communicate with each other using a selected packet-switching network communication protocol. One exemplary communication protocol is the Ethernet communication protocol that enables communication using the Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the Ethernet standard titled "IEEE 802.3 Standard" issued in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of this standard. Alternatively or additionally, the communication devices can communicate with each other using the X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with the standards published by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, the communication devices can communicate with each other using the frame relay communication protocol. The frame relay communication protocol may conform to or be compatible with the standards published by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be able to communicate with each other using the Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard published by the ATM Forum under the title "ATM-MPLS Network Interworking 2.0" in August 2001 and / or later versions of this standard. Naturally, different and / or later-developed connection-oriented network communication protocols are equally contemplated herein.

[0258] Unless otherwise explicitly defined, as will be apparent from the foregoing disclosure, throughout the foregoing disclosure, the use of terms such as "processing", "computing", "calculating", "determining", "displaying", etc. refers to actions and processes of a computer system or similar electronic computing device that operate on and transform data represented as a physical (electronic) quantity in the registers and memories of the computer system into other data similarly represented as a physical quantity in the memory or registers of the computer system or other such information storage, transmission, or display device.

[0259] One or more components may be referred to herein as "configured to", "configurable to", "operable / operative to", "adapted / adaptable", "able to", "conformable / conformed to", etc. Those skilled in the art will understand that "configured to" can generally include components in an active state and / or components in a non-active state and / or components in a standby state, unless the context dictates otherwise.

[0260] The terms "proximal" and "distal" are used herein with reference to a clinician who operates the handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician, and the term "distal" refers to the portion located farther from the clinician. For convenience and clarity, it will be further understood that spatial terms such as "vertical", "horizontal", "up", and "down" may be used herein with respect to the drawings. However, the surgical instrument is used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0261] Those skilled in the art will generally understand that the terms used herein, and particularly those used in the appended claims (e.g., the body of the appended claims), are generally intended to be "open" terms (e.g., the term "including" should be construed as "including but not limited to", the term "having" should be construed as "having at least", the term "includes" should be construed as "includes but is not limited to", etc.). Further, where a particular number is intended in an introduced claim recitation, such intent will be clearly recited in the claim, and those skilled in the art will understand that where there is no such recitation, there is no such intent. For example, by way of illustration, the following appended claims may include introductory phrases such as "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as suggesting that any particular claim including such an introduced claim recitation is limited to a claim having only one such recited item, even where the introduced claim recitation and the indefinite article "a" or "an" are included in the same claim (e.g., "a" and / or "an" should generally be construed as meaning "at least one" or "one or more"). The same holds true when introducing claim recitations using definite articles.

[0262] In addition, even when a specific number is specified in the introduced claim description, those skilled in the art will recognize that such description should typically be interpreted to mean at least the recited number (for example, when there is a mere recitation of "two recitations" without other modifiers, it typically means at least two recitations, or two or more recitations). Further, when a notation similar to "at least one of A, B, and C, etc." is used, generally, such syntax is intended in the sense that those skilled in the art will understand the notation (for example, "a system having at least one of A, B, and C" includes, without limitation, a system having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). When a notation similar to "at least one of A, B, or C, etc." is used, generally, such syntax is intended in the sense that those skilled in the art will understand the notation (for example, "a system having at least one of A, B, or C" includes, without limitation, a system having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C). Further, typically, any disjunctive word and / or phrase representing two or more alternative terms should be understood to be intended to include one of those terms, any of those terms, or both of those terms, whether in the specification, in the claims, or in the drawings, unless the context requires otherwise. For example, the phrase "A or B" will typically be understood to include the possibilities of "A" or "B" or "A and B".

[0263] Regarding the appended claims, those skilled in the art will understand that the recited operations herein can generally be performed in any order. Also, although the flowcharts of various operations are shown in sequence, it should be understood that the various operations may be performed in an order other than that shown, or may be performed simultaneously. Examples of such alternative orderings may include, unless the context dictates otherwise, repetition, interleaving, interruption, reordering, incremental, preparatory, additional, simultaneous, reverse, or other different orderings. Further, terms such as "responsive to", "associated with", or other past-tense adjectives are not generally intended to exclude such variations, unless the context dictates otherwise.

[0264] It is particularly noted that any reference to "one aspect", "aspect", "exemplification", "an exemplification", etc. means that the particular mechanism, structure, or characteristic described in relation to that aspect is included in at least one aspect. Thus, the phrases "in one aspect", "in an aspect", "in an exemplification", and "in an exemplification" that appear in various places throughout this specification do not necessarily all refer to the same aspect. Further, a particular feature, structure, or characteristic can be combined in any suitable manner in one or more aspects.

[0265] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any application data sheet is incorporated herein by reference to the extent that the incorporated material is not inconsistent with this specification. In itself, and to the extent necessary, the disclosure clearly set forth herein shall supersede any conflicting description incorporated herein by reference. Although it is referred to as being incorporated herein by reference, any content, or portions thereof, that are inconsistent with the current definitions, views, or other disclosure content set forth herein shall be incorporated only to the extent that no conflict arises between the incorporated content and the current disclosure content.

[0266] In summary, many benefits resulting from using the concepts described herein have been described. The above description in one or more forms is presented for purposes of illustration and explanation. It is not intended to be exhaustive or to limit to the precise forms disclosed. Modifications or variations are possible in light of the above teachings. One or more forms are selected and described to illustrate the principles and practical applications, thereby enabling one of ordinary skill in the art to utilize the various forms in various modifications as suitable for the particular uses contemplated. The claims presented with this specification are intended to define the overall scope.

[0267] 〔Embodiment〕 (1) A control system for a surgical instrument, the control system comprising: An imaging system comprising: An emitter configured to emit electromagnetic radiation; and An image sensor configured to receive the reflected electromagnetic radiation indicative of a first tissue parameter; A sensor configured to detect a second tissue parameter of tissue acted upon by the surgical instrument; A control circuit coupled to the imaging system and the sensor, the control circuit being configured to: Determine a state of the tissue based on a combination of the first tissue parameter and the second tissue parameter; and Control the surgical instrument according to the determined state of the tissue. (2) The control system according to Embodiment 1, wherein the first tissue parameter includes at least one of an EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift of the tissue. (3) The control system according to Embodiment 1, wherein the second tissue parameter includes at least one of a capacitance or an impedance of the tissue. (4) The control system according to Embodiment 1, wherein the second tissue parameter includes at least one of the temperature, viscoelastic compression, or thickness of the tissue. (5) The control system according to Embodiment 1, wherein the surgical instrument is selected from the group consisting of a surgical stapler, an electrosurgical instrument, and an ultrasonic surgical instrument.

[0268] (6) A control system for a surgical instrument, the surgical instrument including a sensor configured to detect a second parameter associated with the surgical instrument, the control system comprising: An imaging system, comprising: An emitter configured to emit electromagnetic radiation; and An image sensor configured to receive the reflected electromagnetic radiation indicative of a first parameter of the tissue, A control circuit coupled to the imaging system and connectable to the sensor, configured to determine a state of the tissue based on a combination of the first parameter and the second parameter, and configured to control the surgical instrument according to the determined state of the tissue. (7) The control system according to Embodiment 6, wherein the first parameter includes at least one of the EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift of the tissue. (8) The control system according to Embodiment 6, wherein the second parameter includes at least one of the capacitance or impedance of the tissue. (9) The control system according to Embodiment 6, wherein the second parameter includes at least one of the temperature, viscoelastic compression, or thickness of the tissue. (10) The control system according to Embodiment 6, wherein the surgical instrument is selected from the group consisting of a surgical stapler, an electrosurgical instrument, and an ultrasonic surgical instrument.

[0269] (11) The second parameter is the control system according to Embodiment 6, which is associated with a surgical device associated with the surgical instrument. (12) The surgical device includes a surgical generator to which the surgical instrument can be coupled, and is the control system according to Embodiment 11. (13) The second parameter includes the power level of the surgical generator, and is the control system according to Embodiment 12. (14) An imaging system, a surgical device, and a surgical hub connectable to the surgical instrument, wherein the imaging system includes an emitter configured to emit electromagnetic radiation at a surgical site and an image sensor configured to receive the electromagnetic radiation reflected from the surgical site, the reflected electromagnetic radiation indicating a first parameter associated with the tissue, the surgical device including a sensor configured to detect a second parameter associated with the surgical device, and the surgical hub is a control circuit, configured to receive a first measurement of the first parameter from the imaging system, receive a second measurement of the second parameter from the sensor, determine the state of the tissue based on a combination of the first parameter and the second parameter, and control the surgical instrument according to the determined state of the tissue, and includes a control circuit. (15) The first parameter includes at least one of the EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift of the tissue, and is the surgical hub according to Embodiment 14.

[0270] (16) The second parameter includes at least one of the capacitance or impedance of the tissue, and is the surgical hub according to Embodiment 14. (17) The second parameter includes at least one of the temperature, viscoelastic compression, or thickness of the tissue, and is the surgical hub according to Embodiment 14. (18) The surgical hub according to embodiment 14, wherein the surgical instrument is selected from the group consisting of a surgical stapler, an electrosurgical instrument, and an ultrasonic surgical instrument. (19) The surgical hub according to embodiment 14, wherein the surgical device includes a surgical generator to which the surgical instrument can be coupled. (20) The surgical hub according to embodiment 19, wherein the second parameter includes the power level of the surgical generator.

Claims

1. A control system for a surgical instrument, the control system comprising: An imaging system, comprising: An emitter configured to emit electromagnetic radiation; and An image sensor configured to receive the reflected electromagnetic radiation indicative of a first tissue parameter of the tissue acted upon by the surgical instrument, A sensor configured to detect a second tissue parameter of the tissue acted upon by the surgical instrument, the second tissue parameter including a physical parameter, an electrical parameter, or a combination thereof, A control circuit coupled to the imaging system and the sensor, the surgical instrument comprising the control circuit, the control circuit being configured to: Detect the temperature of the tissue based on the second tissue parameter while the tissue is being acted upon by the surgical instrument; Determine that the tissue is undergoing incidental thermal damage based on the first tissue parameter; and Control the surgical instrument acting on the tissue based on the detected temperature of the tissue and the determined incidental thermal damage to the tissue. A control system configured to determine that the tissue is undergoing thermal damage based on the determined incidental thermal damage to the tissue and the detected increase in temperature of the tissue, and to control the surgical instrument based on the determination of the thermal damage to reduce the instrument power level or provide a proposal to the user to reduce the instrument power level.

2. The control system according to claim 1, wherein the surgical instrument is selected from the group consisting of a monopolar probe, a bipolar probe, an ablation probe, and an ultrasonic end effector.

3. The first tissue parameter includes at least one of the EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift of the tissue, and the control system is configured to determine that the tissue is undergoing the thermal damage associated with the action of the surgical instrument by detecting a change in the first tissue parameter. The control system according to claim 2.

4. The control system according to claim 3, wherein the second tissue parameter includes at least one of the capacitance or impedance of the tissue.

5. A control system for a surgical instrument, the surgical instrument including a sensor configured to detect a second parameter associated with the surgical instrument while acting on tissue, the control system An imaging system, An emitter configured to emit electromagnetic radiation, and An imaging sensor configured to receive reflected electromagnetic radiation indicative of a first parameter of the tissue acted upon by the surgical instrument, the imaging system including A control circuit coupled to the imaging system and couplable to the sensor, To detect the temperature of the tissue based on the second parameter while the tissue is being acted upon by the surgical instrument, To determine that the tissue is undergoing incidental thermal damage based on the first parameter, and A control circuit configured to control the surgical instrument acting on the tissue based on the detected temperature of the tissue and the determined incidental thermal damage of the tissue, including Based on the determined incidental thermal damage of the tissue and the detected increase in temperature of the tissue, determine that the tissue is undergoing thermal damage, and based on the determination of the thermal damage, be configured to control the surgical instrument to reduce the instrument power level or provide a proposal to the user to reduce the instrument power level. A control system. **Claim 6** The control system according to claim 5, wherein the surgical instrument is selected from the group consisting of a monopolar probe, a bipolar probe, an ablation probe, and an ultrasonic end effector. **Claim 7** The control system according to claim 6, wherein the second parameter is associated with a surgical device associated with the surgical instrument. **Claim 8** The control system according to claim 7, wherein the surgical device includes a surgical generator to which the surgical instrument is couplable, and the surgical generator is an RF generator or an ultrasonic generator. **Claim 9** The control system according to claim 8, wherein the second parameter includes the power level of the surgical generator. **Claim 10** The first parameter includes at least one of the EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift of the tissue, and the control system is configured to determine that the tissue is undergoing the thermal damage associated with the action of the surgical instrument by detecting a change in the first parameter. The control system according to claim 8.

11. The control system according to claim 9, wherein the second parameter includes at least one of the capacitance or impedance of the tissue.

12. An imaging system, a surgical device, and a surgical hub connectable to a surgical instrument, the imaging system including an emitter configured to emit electromagnetic radiation at a surgical site and an image sensor configured to receive the electromagnetic radiation reflected from the surgical site, the reflected electromagnetic radiation indicating a first parameter associated with the tissue, the surgical device including a sensor configured to detect a second parameter associated with the surgical device, the surgical hub including a control circuit, configured to receive a first measurement of the first parameter from the imaging system, configured to receive a second measurement of the second parameter from the sensor, configured to detect the temperature of the tissue based on the second parameter while the tissue is being acted upon by the surgical instrument, configured to determine that the tissue is undergoing incidental thermal damage based on the first parameter, and configured to control the surgical instrument acting on the tissue based on the detected temperature of the tissue and the determined incidental thermal damage of the tissue. The control circuit includes Based on the determined incidental thermal damage of the tissue and the detected increase in the temperature of the tissue, it is determined that the tissue is undergoing thermal damage, and based on the determination of the thermal damage, the surgical instrument is controlled to reduce the instrument power level or provide a proposal to the user to reduce the instrument power level. A surgical hub configured as such.

13. The surgical hub according to claim 12, wherein the surgical instrument is selected from the group consisting of a monopolar probe, a bipolar probe, an ablation probe, and an ultrasonic end effector.

14. The surgical hub according to claim 13, wherein the surgical device includes a surgical generator to which the surgical instrument is connectable, and the surgical generator is an RF generator or an ultrasonic generator.

15. The surgical hub according to claim 14, wherein the second parameter includes the power level of the surgical generator.

16. The surgical hub according to claim 14, wherein the first parameter includes at least one of the EMR refractive index, EMR polarization, passive IR radiation, or Doppler wavelength shift of the tissue, and the surgical hub is configured to determine that the tissue has received the thermal damage associated with the action of the surgical instrument by detecting a change in the first parameter.

17. The surgical hub according to claim 16, wherein the second parameter includes at least one of the capacitance or impedance of the tissue.

Images