Adaptive surgical system control based on surgical smoke particulate characteristics

Abstract

Various control systems are disclosed for controlling a surgical system according to detected airborne particle characteristics. In particular, the present disclosure relates to a control system for a surgical system, the control system including an imaging system including an emitter configured to emit electromagnetic radiation (EMR), a polarizing filter configured to polarize the EMR emitted by the emitter, an image sensor, and control circuitry coupled to the imaging system, the control circuitry configured to detect polarized EMR reflected from the airborne particle received by the image sensor, determine a particle type of the airborne particle based on the reflectance characteristics of the polarized EMR, whether the particle type is a naturally occurring particle or a synthetic particle, and adjust control parameters of the surgical system according to the determined particle type.

Description

[Background technology]

[0001] Surgical systems often incorporate imaging systems that can enable the clinician to view the surgical site and / or one or more portions thereof on one or more displays, such as, for example, a monitor. The displays may be local and / or remote to the operating room. The imaging system may include a scope with a camera that views the surgical site and transmits images to a display that can be viewed by the clinician. Scopes include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-pelvic, sigmoidoscopes, thoracoscopes, ureteroscopes, and exoscopes. Imaging systems may be limited by the information they can recognize and / or communicate to the clinician. For example, certain hidden structures, physical contours, and / or dimensions in three-dimensional space may not be recognized intraoperatively by certain imaging systems. In addition, certain imaging systems may not be able to communicate and / or communicate certain information to the clinician intraoperatively. Summary of the Invention [Means for solving the problem]

[0002] In one general aspect, a control system for a surgical system is disclosed. The control system includes an imaging system and a control circuit coupled to the imaging system. The imaging system includes an emitter configured to emit electromagnetic radiation (EMR), a polarizing filter configured to polarize the EMR emitted by the emitter, and an image sensor. The control circuit is configured to detect polarized EMR reflected from an airborne particle received by the image sensor, determine a particulate type of the airborne particle as a naturally occurring particulate or a synthetic particulate based on a reflectance characteristic of the polarized EMR, and adjust control parameters of the surgical system according to the determined particulate type.

[0003] In another general aspect, a control system for a surgical system is disclosed. The control system includes an imaging system and a control circuit coupled to the imaging system. The imaging system includes a multispectral electromagnetic radiation (EMR) source, a polarizing filter configured to polarize the EMR emitted by the multispectral EMR source, and an image sensor. The control circuit is configured to detect polarized EMR reflected from a particulate in an aerosol received by the image sensor, determine whether a particulate type of the particulate is a naturally occurring particulate or a synthetic particulate based on a reflectance characteristic of the polarized EMR, determine whether to adjust a first control parameter or a second control parameter according to the determined particulate type, and adjust the first control parameter or the second control parameter accordingly.

[0004] In yet another general aspect, a control system for a surgical system is disclosed. The control system includes an imaging system and a control circuit coupled to the imaging system. The imaging system includes a multispectral electromagnetic radiation (EMR) source, a polarizing filter configured to polarize the EMR emitted by the multispectral EMR source, and an image sensor. The control circuit is configured to detect polarized EMR reflected from a particulate in an aerosol received by the image sensor, determine whether a particulate type of the particulate is a naturally occurring particulate or a synthetic particulate based on a reflectance characteristic of the polarized EMR, determine whether to adjust a first control parameter or a second control parameter according to the determined particulate type, and adjust the first control parameter or the second control parameter accordingly. [Brief description of the drawings]

[0005] The novel features of the various aspects are set forth with particularity in the appended claims, however the described aspects, both as to organization and method of operation, may best be understood by reference to the following description taken in conjunction with the accompanying drawings, in which: [Figure 1]FIG. 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, in accordance with at least one embodiment of the present disclosure, the surgical visualization system configured to identify critical structures beneath a tissue surface. [Diagram 2] FIG. 1 is a schematic diagram of a control system for a surgical visualization system, according to at least one embodiment of the present disclosure. [Figure 2A] 1 illustrates a control circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Figure 2B] 1 illustrates a combinatorial logic circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Figure 2C] 1 illustrates a sequential logic circuit configured to control aspects of a surgical visualization system in accordance with at least one aspect of the present disclosure. [Diagram 3] 2 is a schematic diagram illustrating triangulation between the surgical device, imaging device, and critical structure of FIG. 1 to determine the depth dA of the critical structure below the tissue surface, in accordance with at least one embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a surgical visualization system configured to identify a critical structure below a tissue surface, in accordance with at least one embodiment of the present disclosure, the surgical visualization system including a pulsed light source for determining a depth dA of the critical structure below the tissue surface. [Diagram 5] FIG. 1 is a schematic diagram of a surgical visualization system including an imaging device and a surgical device, in accordance with at least one embodiment of the present disclosure, the surgical visualization system configured to identify critical structures beneath a tissue surface. [Figure 6] FIG. 1 is a schematic diagram of a surgical visualization system including a three-dimensional camera, the surgical visualization system configured to identify critical structures embedded within tissue, in accordance with at least one embodiment of the present disclosure. [Figure 7A] 7A and 7B are images of a structure of interest taken with the 3D camera of FIG. 6, in accordance with at least one embodiment of the present disclosure, where FIG. 7A is an image from the left lens of the 3D camera and FIG. 7B is an image from the right lens of the 3D camera. [Figure 7B] 7A and 7B are images of a structure of interest taken with the 3D camera of FIG. 6, in accordance with at least one embodiment of the present disclosure, where FIG. 7A is an image from the left lens of the 3D camera and FIG. 7B is an image from the right lens of the 3D camera. [Figure 8] FIG. 7 is a schematic diagram of the surgical visualization system of FIG. 6 capable of determining a camera-to-critical structure distance dw from a three-dimensional camera to a critical structure in accordance with at least one embodiment of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of a surgical visualization system utilizing two cameras to determine the location of implanted critical structures, in accordance with at least one embodiment of the present disclosure. [Figure 10A] FIG. 1 is a schematic diagram of a surgical visualization system that utilizes a camera that moves axially between multiple known positions to determine the location of an implanted critical structure, in accordance with at least one embodiment of the present disclosure. [Figure 10B] FIG. 10B is a schematic diagram of the surgical visualization system of FIG. 10A in which a camera moves axially and rotationally between a number of known positions to determine the location of an implanted critical structure, in accordance with at least one embodiment of the present disclosure. [Figure 11] FIG. 1 is a schematic diagram of a control system for a surgical visualization system, according to at least one embodiment of the present disclosure. [Figure 12] FIG. 1 is a schematic diagram of a structured light source of a surgical visualization system in accordance with at least one embodiment of the present disclosure. [Figure 13A] 1 is a graph of absorption coefficient versus wavelength for various biomaterials, in accordance with at least one embodiment of the present disclosure. [Figure 13B] FIG. 1 is a schematic diagram of visualization of an anatomical structure via a spectral surgical visualization system, in accordance with at least one embodiment of the present disclosure. [Figure 13C] 13A-13E show exemplary hyperspectral discrimination signatures for distinguishing anatomical structures from occluders in accordance with at least one embodiment of the present disclosure, where FIG. 13C is a graphical representation of a ureteral signature versus occluders, FIG. 13D is a graphical representation of an arterial signature versus occluders, and FIG. 13E is a graphical representation of a neural signature versus occluders. [Figure 13D] 13A-13E show exemplary hyperspectral discrimination signatures for distinguishing anatomical structures from occluders in accordance with at least one embodiment of the present disclosure, where FIG. 13C is a graphical representation of a ureteral signature versus occluders, FIG. 13D is a graphical representation of an arterial signature versus occluders, and FIG. 13E is a graphical representation of a neural signature versus occluders. [Figure 13E] 13A-13E show exemplary hyperspectral discrimination signatures for distinguishing anatomical structures from occluders in accordance with at least one embodiment of the present disclosure, where FIG. 13C is a graphical representation of a ureteral signature versus occluders, FIG. 13D is a graphical representation of an arterial signature versus occluders, and FIG. 13E is a graphical representation of a neural signature versus occluders. [Figure 14] FIG. 1 is a schematic diagram of a near-infrared (NIR) time-of-flight measurement system configured to sense distance to a critical anatomical structure, in accordance with at least one embodiment of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) and a receiver (sensor) disposed on a common device. [Figure 15] 17B is a schematic diagram of an emitted wave, a received wave, and a delay between the emitted wave and the received wave of the NIR time-of-flight measurement system of FIG. 17A in accordance with at least one embodiment of the present disclosure. [Figure 16] 1 illustrates an NIR time-of-flight measurement system configured to sense distance to a critical structure in accordance with at least one embodiment of the present disclosure, the time-of-flight measurement system including a transmitter (emitter) on another device and a receiver (sensor). [Figure 17] FIG. 1 is a block diagram of a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 18] 1 is a surgical system used to perform a surgical procedure in an operating room, according to at least one embodiment of the present disclosure. [Figure 19] 1 illustrates a computer-implemented interactive surgical system according to at least one aspect of the present disclosure. [Figure 20] 1 shows a diagram of a situationally aware surgical system according to at least one embodiment of the present disclosure. [Figure 21] 1 shows a timeline illustrating situational awareness of a hub, in accordance with at least one aspect of the present disclosure. [Figure 22] FIG. 1 illustrates a surgical system during the performance of a surgical procedure, according to at least one embodiment of the present disclosure. [Figure 23] FIG. 1 is a diagram of an imaging device facing multiple obstructions, in accordance with at least one embodiment of the present disclosure. [Figure 24] FIG. 1 is a logic flow diagram of a process for generating a fusion image utilizing a multispectral EMR source, in accordance with at least one embodiment of the present disclosure. [Diagram 25] FIG. 2 is a diagram of a fusion image generated from a multispectral EMR source, in accordance with at least one embodiment of the present disclosure. [Figure 26] FIG. 2 is a logic flow diagram of a process for generating a fused image utilizing multiple image frames in accordance with at least one aspect of the present disclosure. [Figure 27] 1 is a diagram of a sequence of image frames in accordance with at least one aspect of the present disclosure. [Figure 28] FIG. 1 is a diagram of a fusion image, in accordance with at least one embodiment of the present disclosure. [Figure 29] FIG. 1 is a diagram of a fusion image visualized to a user, in accordance with at least one aspect of the present disclosure. [Diagram 30] FIG. 1 is a schematic diagram of a surgical instrument according to at least one embodiment of the present disclosure. [Diagram 31] FIG. 13 is a logic flow diagram of a process for controlling a surgical system based on a plurality of sensed parameters, in accordance with at least one aspect of the present disclosure. [Diagram 32] FIG. 2 is a diagram of a polarized EMR source for detecting different particulate types, in accordance with at least one embodiment of the present disclosure. [Figure 33A] FIG. 13 is a logic flow diagram of a process for controlling a surgical system according to a type of particulate detected, in accordance with at least one embodiment of the present disclosure. [Figure 33B]FIG. 13 is a logic flow diagram of a process for controlling a surgical system according to a type of detected particulate detected within a defined range gate, in accordance with at least one aspect of the present disclosure. [Figure 34A] 1 is a pixel array of an image sensor for detecting airborne particulates, according to at least one embodiment of the present disclosure. [Figure 34B] 34B is a pixel array of an image sensor detecting an airborne particle that has moved from the position shown in FIG. 34A, according to at least one embodiment of the present disclosure. [Figure 34C] 34C is a pixel array of an image sensor illustrating a generalized motion vector of the particulate matter shown in FIG. 34B, in accordance with at least one embodiment of the present disclosure. [Diagram 35] 34A-34C show changes in suspended particulate cloud conditions, according to at least one embodiment of the present disclosure. [Diagram 36] FIG. 1 is a diagram of a surgical system during the performance of a surgical procedure in which a particulate cloud is being generated, in accordance with at least one embodiment of the present disclosure. [Figure 37] FIG. 13 is a logic flow diagram of a process for controlling a surgical system according to particulate cloud characteristics, in accordance with at least one embodiment of the present disclosure. [Figure 38] 1 is a series of graphs illustrating adjustment of control parameters based on particulate cloud characteristics in accordance with at least one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The applicant of the present application owns the following concurrently filed US patent applications, the entire contents of each of which are incorporated herein by reference: Attorney Docket No. END9228USNP1 / 190580-1M, Title of invention: "METHOD OF USING IMAGING DEVICES IN SURGERY" -Attorney docket number END9227USNP1 / 190579-1, invention title "ADAPTIVE VISUALIZATION BY A SURGICAL SYSTEM", -Attorney docket number END9226USNP1 / 190578-1, invention title "SURGICAL SYSTEM CONTROL BASED ON MULTIPLE SENSED PARAMETERS", -Attorney Docket No. END9225USNP1 / 190577-1, Invention Title: "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE PARTICLE CHARACTERISTICS"; -Attorney docket number END9224USNP1 / 190576-1, invention title "ADAPTIVE SURGICAL SYSTEM CONTROL ACCORDING TO SURGICAL SMOKE CLOUD CHARACTERISTICS", -Attorney docket number END9223USNP1 / 190575-1, invention title "SURGICAL SYSTEMS CORRELATING VISUALIZATION DATA AND POWERED SURGICAL INSTRUMENT DATA", -Attorney Reference Number END9222USNP1 / 190574-1, Title of Invention: "SURGICAL SYSTEMS FOR GENERATING THREE DIMENSIONAL CONSTRUCTS OF ANATOMICAL ORGANS AND COUPLING IDENTIFIED" -Attorney docket number END9221USNP1 / 190573-1, invention title "SURGICAL SYSTEM FOR OVERLAYING SURGICAL INSTRUMENT DATA ONTO A VIRTUAL THREE Dimensional Construct of an Organ", -Attorney docket number END9220USNP1 / 190572-1, invention title "SURGICAL SYSTEMS FOR PROPOSING AND CORROBORATING ORGAN PORTION REMOVALS", -Attorney Reference Number END9219USNP1 / 190571-1, Title of Invention "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE", -Attorney docket number END9218USNP1 / 190570-1, invention title "VISUALIZATION SYSTEMS USING STRUCTURED LIGHT", Attorney Docket No. END9217USNP1 / 190569-1, Title of Invention: "DYNAMIC SURGICAL VISUALIZATION SYSTEMS", and -Attorney Docket No. END9216USNP1 / 190568-1, invention title "ANALYZING SURGICAL TRENDS BY A SURGICAL SYSTEM".

[0007] The applicant of this application owns the following U.S. patent applications, filed on March 15, 2019, the entire contents of each of which are incorporated herein by reference: - U.S. Patent Application No. 16 / 354,417, entitled "INPUT CONTROLS FOR ROBOTIC SURGERY"; - U.S. Patent Application No. 16 / 354,420, entitled "DUAL MODE CONTROLS FOR ROBOTIC SURGERY"; - U.S. Patent Application No. 16 / 354,422, entitled "MOTION CAPTURE CONTROLS FOR ROBOTIC SURGERY"; - U.S. Patent Application No. 16 / 354,440, entitled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING SURGICAL TOOL MOTION ACCORDING TO TISSUE PROXIMITY"; - U.S. Patent Application No. 16 / 354,444, entitled "ROBOTIC SURGICAL SYSTEMS WITH MECHANISMS FOR SCALING CAMERA MAGNIFICATION ACCORDING TO PROXIMITY OF SURGICAL TOOL TO TISSUE"; - U.S. Patent Application No. 16 / 354,454, entitled "ROBOTIC SURGICAL SYSTEMS WITH SELECTIVELY LOCKABLE END EFFECTORS"; - U.S. Patent Application No. 16 / 354,461, entitled "SELECTABLE VARIABLE RESPONSE OF SHAFT MOTION OF SURGICAL ROBOTIC SYSTEMS"; - U.S. Patent Application No. 16 / 354,470, entitled "SEGMENTED CONTROL INPUTS FOR SURGICAL ROBOTIC SYSTEMS"; - U.S. Patent Application No. 16 / 354,474, entitled "ROBOTIC SURGICAL CONTROLS HAVING FEEDBACK CAPABILITIES"; U.S. Patent Application No. 16 / 354,478, entitled "ROBOTIC SURGICAL CONTROLS WITH FORCE FEEDBACK"; and · U.S. Patent Application No. 16 / 354,481 entitled "JAW COORDINATION OF ROBOTIC SURGICAL CONTROLS"

[0008] The applicant of this application also owns the following U.S. patent applications, filed on September 11, 2018, the entire contents of each of which are incorporated herein by reference: - U.S. Patent Application No. 16 / 128,179, entitled "SURGICAL VISUALIZATION PLATFORM"; U.S. Patent Application No. 16 / 128,180, entitled "CONTROLLING AN EMITTER ASSEMBLY PULSE SEQUENCE"; - U.S. Patent Application No. 16 / 128,198, entitled "SINGULAR EMR SOURCE EMITTER ASSEMBLY"; - U.S. Patent Application No. 16 / 128,207, entitled "COMBINATION EMITTER AND CAMERA ASSEMBLY"; - U.S. Patent Application No. 16 / 128,176, entitled "SURGICAL VISUALIZATION WITH PROXIMITY TRACKING FEATURES"; - U.S. Patent Application No. 16 / 128,187, entitled "SURGICAL VISUALIZATION OF MULTIPLE TARGETS"; · U.S. Patent Application No. 16 / 128,192, entitled "VISUALIZATION OF SURGICAL DEVICES"; - U.S. Patent Application No. 16 / 128,163, entitled "OPERATIVE COMMUNICATION OF LIGHT"; - U.S. Patent Application No. 16 / 128,197, entitled "ROBOTIC LIGHT PROJECTION TOOLS"; - U.S. Patent Application No. 16 / 128,164, entitled "SURGICAL VISUALIZATION FEEDBACK SYSTEM"; - U.S. Patent Application No. 16 / 128,193, entitled "SURGICAL VISUALIZATION AND MONITORING"; - U.S. Patent Application No. 16 / 128,195, entitled "INTEGRATION OF IMAGING DATA"; - U.S. Patent Application No. 16 / 128,170, entitled "ROBOTICALLY-ASSISTED SURGICAL SUTURING SYSTEMS"; - U.S. Patent Application No. 16 / 128,183, entitled "SAFETY LOGIC FOR SURGICAL SUTURING SYSTEMS"; U.S. Patent Application No. 16 / 128,172, entitled "ROBOTIC SYSTEM WITH SEPARATE PHOTOACOUSTIC RECEIVER"; and - U.S. Patent Application No. 16 / 128,185 entitled "FORCE SENSOR THROUGH STRUCTURED LIGHT DEFLECTION"

[0009] The applicant of this application also owns the following U.S. patent applications, filed on March 29, 2018, the entire contents of each of which are incorporated herein by reference: U.S. Patent Application No. 15 / 940,627, entitled "DRIVE ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS," now U.S. Patent Application Publication No. 2019 / 0201111; U.S. Patent Application No. 15 / 940,676, entitled "AUTOMATIC TOOL ADJUSTMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS," now U.S. Patent Application Publication No. 2019 / 0201142; U.S. Patent Application No. 15 / 940,711, entitled "SENSING ARRANGEMENTS FOR ROBOT-ASSISTED SURGICAL PLATFORMS," now U.S. Patent Application Publication No. 2019 / 0201120; and U.S. Patent Application No. 15 / 940,722, entitled "CHARACTERIZATION OF TISSUE IRREGULARITIES THROUGH THE USE OF MONO-CHROMATIC LIGHT REFRACTIVITY," now U.S. Patent Application Publication No. 2019 / 0200905.

[0010] The applicant of the present application owns the following U.S. patent applications, filed on December 4, 2018, the disclosures of each of which are incorporated by reference in their entirety herein: No. 16 / 209,395, entitled "METHOD OF HUB COMMUNICATION", now U.S. Patent Application Publication No. 2019 / 0201136; U.S. Patent Application No. 16 / 209,403, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," now U.S. Patent Application Publication No. 2019 / 0206569; No. 16 / 209,407, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL", now U.S. Patent Application Publication No. 2019 / 0201137; U.S. Patent Application No. 16 / 209,416, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS," now U.S. Patent Application Publication No. 2019 / 0206562; U.S. Patent Application No. 16 / 209,423, entitled "METHOD OF COMPRESSING TISSUE WITHIN A STAPLING DEVICE AND SIMULTANEOUSLY DISPLAYING THE LOCATION OF THE TISSUE WITHIN THE JAWS", now U.S. Patent Application Publication No. 2019 / 0200981; U.S. Patent Application No. 16 / 209,427, entitled "METHOD OF USING REINFORCED FLEXIBLE CIRCUITS WITH MULTIPLE SENSORS TO OPTIMIZE PERFORMANCE OF RADIO FREQUENCY DEVICES," now U.S. Patent Application Publication No. 2019 / 0208641; U.S. Patent Application No. 16 / 209,433, entitled "METHOD OF SENSING PARTICULATE FROM SMOKE EVACUATED FROM A PATIENT, ADJUSTING THE PUMP SPEED BASED ON THE SENSED INFORMATION, AND COMMUNICATING THE FUNCTIONAL PARAMETERS OF THE SYSTEM TO THE HUB", now U.S. Patent Application Publication No. 2019 / 0201594; U.S. Patent Application No. 16 / 209,447, entitled "METHOD FOR SMOKE EVACUATION FOR SURGICAL HUB", now U.S. Patent Application Publication No. 2019 / 0201045; U.S. Patent Application No. 16 / 209,453, entitled "METHOD FOR CONTROLLING SMART ENERGY DEVICES," now U.S. Patent Application Publication No. 2019 / 0201046; U.S. Patent Application No. 16 / 209,458, entitled "METHOD FOR SMART ENERGY DEVICE INFRASTRUCTURE", now U.S. Patent Application Publication No. 2019 / 0201047; U.S. Patent Application No. 16 / 209,465, entitled "METHOD FOR ADAPTIVE CONTROL SCHEMES FOR SURGICAL NETWORK CONTROL AND INTERACTION," now U.S. Patent Application Publication No. 2019 / 0206563; U.S. Patent Application No. 16 / 209,478, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE," now U.S. Patent Application Publication No. 2019 / 0104919; U.S. Patent Application No. 16 / 209,490, entitled "METHOD FOR FACILITY DATA COLLECTION AND INTERPRETATION," now U.S. Patent Application Publication No. 2019 / 0206564; and U.S. Patent Application No. 16 / 209,491, entitled "METHOD FOR CIRCULAR STAPLER CONTROL ALGORITHM ADJUSTMENT BASED ON SITUATIONAL AWARENESS," now U.S. Patent Application Publication No. 2019 / 0200998.

[0011] Before describing the various aspects of the surgical visualization platform in detail, it should be noted that the example embodiments are not limited in application or use to the details of construction and arrangement of parts illustrated in the accompanying drawings and specification. The example embodiments may be embodied or incorporated in other aspects, variations, and modifications and may be practiced or carried out in various ways. Moreover, unless otherwise specified, the terms and phrases used herein have been selected for the convenience of the reader for the purpose of describing the example embodiments and not for the purpose of limiting them. Furthermore, it should be understood that one or more of the aspects, aspect expressions, and / or examples described below can be combined with any one or more of the other aspects, aspect expressions, and / or examples described below.

[0012] Surgical Visualization Systems The present disclosure is directed to a surgical visualization platform that leverages "digital surgery" to obtain additional information regarding a patient's anatomy and / or a surgical procedure. The surgical visualization platform is further configured to communicate data and / or information in a useful manner to one or more clinicians. For example, various aspects of the present disclosure provide improved visualization of a patient's anatomy and / or a surgical procedure.

[0013] "Digital surgery" may encompass robotic systems, advanced imaging, advanced instruments, artificial intelligence, machine learning, data analytics for performance tracking and benchmarking, connectivity both inside and outside the operating room (OR), and the like. Although the various surgical visualization platforms described herein can be used in combination with robotic surgical systems, the surgical visualization platforms are not limited to use with robotic surgical systems. In certain examples, advanced surgical visualization can be performed without a robot and / or with limited and / or optional robotic assistance. Similarly, digital surgery can be performed without a robot and / or with limited and / or optional robotic assistance.

[0014] In certain examples, a surgical system incorporating a surgical visualization platform may enable smart dissection to identify and avoid critical structures. Critical structures include anatomical structures, such as ureters, arteries such as the superior mesenteric artery, veins such as the portal vein, nerves such as the phrenic nerve, and / or tumors, among other anatomical structures. In other examples, critical structures may be foreign structures in the anatomical field, such as surgical devices, surgical fasteners, clips, clasps, bougies, bands, and / or plates. Critical structures may be determined on a patient-by-patient and / or procedure-by-procedure basis. Exemplary critical structures are described further herein. Smart dissection may provide improved intraoperative guidance for dissection and / or enable smarter decisions, for example, through critical anatomical detection and avoidance techniques.

[0015] Surgical systems incorporating surgical visualization platforms may also enable smart anastomosis, providing a more reliable anastomosis at an optimal location with improved workflow. Cancer localization methods may also be improved with the various surgical visualization platforms and procedures described herein. For example, cancer localization methods may identify and track the location, orientation, and margins of the cancer. In certain instances, cancer localization methods may compensate for the movement of instruments, the patient, and / or the patient's anatomy during surgery and redirect the clinician to the point of interest.

[0016] In certain aspects of the present disclosure, the surgical visualization platform may provide improved tissue characterization and / or lymph node diagnosis and mapping. For example, tissue characterization techniques can characterize tissue types and health without the need for physical touch, particularly during incision and / or placement of a stapling device within the tissue. Certain tissue characterization techniques described herein can be utilized without the use of ionizing radiation and / or contrast agents. With respect to lymph node diagnosis and mapping, the surgical visualization platform can preoperatively locate, map, and ideally diagnose the lymphatic system and / or lymph nodes involved in, for example, cancer diagnosis and staging.

[0017] During surgery, information available to the clinician via the "naked eye" and / or imaging system may provide an incomplete view of the surgical site. For example, certain structures, such as structures embedded or covered within an organ, may be at least partially hidden from view, i.e., invisible. In addition, certain dimensions and / or relative distances may be difficult to ascertain with existing sensor systems and / or difficult to grasp with the "naked eye." Furthermore, certain structures may move pre-operatively (e.g., before surgery but after a pre-operative scan) and / or intra-operatively. In such instances, the clinician may not be able to accurately determine the location of critical structures intra-operatively.

[0018] When the location of critical structures is uncertain and / or the proximity between critical structures and surgical tools is unknown, the clinician's decision-making process may be hindered. For example, the clinician may avoid certain areas to avoid inadvertent dissection of critical structures, but the avoided areas may be unnecessarily large and / or at least partially mislocated. Due to uncertainty and / or excessive / overcaution, the clinician may not be able to reach a particular desired area. For example, overcaution may cause the clinician to leave some of the tumor and / or other undesirable tissue behind in an effort to avoid critical structures, even if the critical structures are not in that particular area and / or there is no adverse effect from the clinician's actions in that particular area. In certain instances, surgical outcomes may improve with increased knowledge and / or certainty, which allows the surgeon to be more precise and, in certain cases, more conservative / more aggressive in certain anatomical areas.

[0019] In various aspects, the present disclosure provides a surgical visualization system for intraoperative identification and avoidance of critical structures. In one aspect, the present disclosure provides a surgical visualization system that enables enhanced intraoperative decision making and improved surgical outcomes. In various aspects, the disclosed surgical visualization systems provide advanced visualization capabilities beyond what a clinician sees with the "naked eye" and / or what an imaging system can recognize and / or communicate to a clinician. Various surgical visualization systems can improve outcomes in various instances by being able to augment and enhance what a clinician can know prior to tissue processing (e.g., incision).

[0020] For example, the visualization system may include a first light emitter configured to emit a plurality of spectral waves, a second light emitter configured to emit a light pattern, and one or more receivers or sensors configured to detect visible light, molecular responses to the spectral waves (spectroscopic imaging), and / or the light pattern. It should be noted that throughout the following disclosure, unless specifically mentioned with respect to visible light, any reference to "light" may include EMR or photons in the visible and / or non-visible portions of the electromagnetic radiation (EMR) wavelength spectrum. The surgical visualization system may also include an imaging system and control circuitry in signal communication with the receiver and the imaging system. Based on the output from the receiver, the control circuitry may determine a geometric surface map, i.e., a three-dimensional surface topography, of visible surfaces at the surgical site and one or more distances relative to the surgical site. In certain examples, the control circuitry may determine one or more distances to at least partially hidden structures. Additionally, the imaging system may communicate the geometric surface map and one or more distances to a clinician. In such examples, an augmented view of the surgical site provided to the clinician may provide a view of hidden structures within a context associated with the surgical site. For example, the imaging system can virtually extend hidden structures onto a geometric surface map of the obscuring and / or obstructing tissue, similar to a line drawn on the ground to indicate subsurface utility lines. Additionally or alternatively, the imaging system can communicate the proximity of one or more surgical tools to the visibly obstructing tissue and / or to the at least partially hidden structures, and / or the depth of the hidden structures below the visible surface of the obstructing tissue. For example, the visualization system can determine the distance to the extension line on the surface of the visible tissue and communicate that distance to the imaging system.

[0021] In various aspects of the present disclosure, a surgical visualization system is disclosed for intraoperative identification and avoidance of critical structures. Such a surgical visualization system can provide valuable information to the clinician during a surgical procedure. As a result, the clinician can maintain control throughout the surgery knowing that the surgical visualization system is tracking critical structures, such as, for example, ureters, specific nerves, and / or critical blood vessels, that may be accessible during, for example, dissection. In one aspect, the surgical visualization system can provide instructions to the clinician long enough for the clinician to pause and / or slow down the surgical procedure and evaluate the proximity to critical structures to prevent inadvertent damage to the structures. The surgical visualization system can provide an ideal, optimized, and / or customizable amount of information to the clinician to enable the clinician to control the entire tissue reliably and / or quickly while avoiding inadvertent damage to healthy tissue and / or critical structures, thereby minimizing the risk of injury resulting from the surgical procedure.

[0022] FIG. 1 is a schematic diagram of a surgical visualization system 100 according to at least one embodiment of the present disclosure. The surgical visualization system 100 can generate a visual display of a critical structure 101 within an anatomical field. The surgical visualization system 100 can be used, for example, for clinical analysis and / or medical intervention. In certain examples, the surgical visualization system 100 can be used intraoperatively to provide a clinician with real-time or near real-time information regarding proximity, dimensions, and / or distance during a surgical procedure. The surgical visualization system 100 is configured for intraoperative identification of critical structures and / or to facilitate avoidance of the critical structure 101 by a surgical device. For example, by identifying the critical structure 101, a clinician can avoid manipulating a surgical device around the critical structure 101 and / or a predetermined proximal area of ​​the critical structure 101 during a surgical procedure. A clinician can avoid dissection of, for example, veins, arteries, nerves, and / or blood vessels, and / or adjacent thereto, identified as the critical structure 101. In various examples, the critical structures 101 may be determined on a patient-by-patient and / or procedure-by-procedure basis.

[0023] The surgical visualization system 100 incorporates tissue identification and geometric surface mapping in combination with a distance sensor system 104. The combined features of the surgical visualization system 100 allow for determining the location of the critical structure 101 within the anatomical field and / or the proximity of the surgical device 102 to the visible tissue surface 105 and / or the critical structure 101. Additionally, the surgical visualization system 100 includes an imaging system including an imaging device 120, such as a camera configured to provide real-time images of the surgical site. In various examples, the imaging device 120 is a spectral camera (e.g., a hyperspectral camera, a multispectral camera, or a selective spectrum camera) configured to detect reflected spectral waveforms and generate a spectral cube of images based on molecular responses to different wavelengths. Images from the imaging device 120 can be provided to a clinician and, in various aspects of the present disclosure, additional information can be added based on tissue identification, situation mapping, and the distance sensor system 104. In such examples, the surgical visualization system 100 includes multiple subsystems, namely, an imaging subsystem, a surface mapping subsystem, a tissue identification subsystem, and / or a distance determination subsystem. These subsystems work together to provide advanced data synthesis and integrated information to the clinician intraoperatively.

[0024] The imaging device may include, for example, a camera or imaging sensor configured to detect visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). In various aspects of the present disclosure, the imaging system may include an imaging device, such as, for example, an endoscope. Additionally or alternatively, the imaging system may include an imaging device, such as, for example, an arthroscope, angioscope, bronchoscope, cholangioscope, colonoscope, cystoscope, duodenoscope, enteroscope, esophagogastroduodenoscope (gastroscope), laryngoscope, nasopharyngo-neproscope, sigmoidoscope, thoracoscope, ureteroscope, or exoscope. In other examples, such as open surgery applications, the imaging system may not include a scope.

[0025] In various aspects of the present disclosure, the tissue identification subsystem can be accomplished with a spectral imaging system. The spectral imaging system can rely on, for example, hyperspectral imaging, multispectral imaging, or selective spectral imaging. Hyperspectral imaging of tissue is further described in U.S. Patent No. 9,274,047, issued March 1, 2016, entitled "SYSTEM AND METHOD FOR GROSS ANATOMIC PATHOLOGY USING HYPERSPECTRAL IMAGING," which is incorporated herein by reference in its entirety.

[0026] In various aspects of the present disclosure, the surface mapping subsystem can be accomplished with a light pattern system, as further described herein. The use of light patterns (or structured light) for surface mapping is known. Known surface mapping techniques can be utilized in the surgical visualization systems described herein.

[0027] Structured light is the process of projecting a known pattern, often a grid or horizontal bars, onto a surface. U.S. Patent Application Publication No. 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled "DEPICTION SYSTEM," published on September 7, 2017, disclose a surgical system that includes a light source and a projector for projecting a light pattern. U.S. Patent Application Publication No. 2017 / 0055819, entitled "SET COMPRISING A SURGICAL INSTRUMENT," published on March 2, 2017, and U.S. Patent Application Publication No. 2017 / 0251900, entitled "DEPICTION SYSTEM," published on September 7, 2017, are incorporated herein by reference in their respective entireties.

[0028] In various aspects of the present disclosure, a distance determination system can be incorporated into a surface mapping system. For example, structured light can be utilized to generate a three-dimensional virtual model of the visible surface and determine various distances to the visible surface. Additionally or alternatively, the distance determination system can rely on time-of-flight measurements to determine one or more distances to identified tissues (or other structures) at the surgical site.

[0029] 2 is a schematic diagram of a control system 133 that may be utilized with surgical visualization system 100. Control system 133 includes control circuitry 132 in signal communication with memory 134. Memory 134 stores instructions executable by control circuitry 132 to determine and / or recognize structures of interest (e.g., structure of interest 101 of FIG. 1), determine and / or calculate one or more distances and / or three-dimensional digital representations, and communicate certain information to one or more clinicians. For example, memory 134 stores surface mapping logic 136, imaging logic 138, tissue identification logic 140, or distance determination logic 141, or any combination of logic 136, 138, 140, and 141. Control system 133 also includes an imaging system 142 having one or more cameras 144 (such as imager 120 of FIG. 1), one or more displays 146, or one or more controls 148, or any combination of these elements. The camera 144 can include one or more image sensors 135 (e.g., visible light, spectral imagers, 3D lenses, among others) for receiving signals from various light sources emitting light in various visible and invisible spectrums. The display 146 can include one or more screens or monitors for portraying real, virtual, and / or virtually augmented images and / or information to one or more clinicians.

[0030] In various embodiments, the heart of the camera 144 is the image sensor 135. Generally, modern image sensors 135 are solid-state electronic devices that contain up to millions of separate light-sensing sites called pixels. Image sensor 135 technology falls into one of two categories: Charge-Coupled Device (CCD) and Complementary Metal Oxide Semiconductor (CMOS) imagers, with more recently, Short Wave Infrared (SWIR) being an emerging technology in imaging. Another type of image sensor 135 employs a hybrid CCD / CMOS architecture (sold under the name "sCOMS") and consists of a CMOS readout integrated circuit (ROIC) bump-bonded to a CCD imaging substrate. CCD and CMOS image sensors 135 have sensitivity to wavelengths from about 350 to 1050 nm, although this range is typically 400 to 1000 nm. CMOS sensors are generally more sensitive to IR wavelengths than CCD sensors. Solid-state image sensors 135 are based on the photoelectric effect and, as a result, cannot distinguish colors. Thus, there are two types of color CCD cameras: 1-chip and 3-chip. 1-chip color CCD cameras provide a common low-cost imaging solution, using a mosaic (e.g., Bayer) optical filter to separate the incoming light into a series of colors and then using an interpolation algorithm to resolve a full-color image. Each color is then directed to a different set of pixels. 3-chip color CCD cameras provide higher resolution by using a prism to direct each section of the incoming spectrum to a different chip. More accurate color reproduction is possible because each point in the space of an object has a separate RGB intensity value, rather than using an algorithm to determine color. 3-chip cameras provide very high resolution.

[0031] The control system 133 also includes a spectral light source 150 and a structured light source 152. In certain examples, a single light source can pulse wavelengths of light within the range of the spectral light source 150 and wavelengths of light within the range of the structured light source 152. Alternatively, a single light source can pulse wavelengths of light within the invisible spectrum (e.g., infrared spectrum light) and on the visible spectrum. The spectral light source 150 can be, for example, a hyperspectral light source, a multispectral light source, and / or a selective spectrum light source. In various examples, the tissue identification logic 140 can identify structures of interest via data from the spectral light source 150 received by the image sensor 135 portion of the camera 144. The surface mapping logic 136 can determine a contour of the surface of the visible tissue based on the reflected structured light. Through time-of-flight measurements, the distance determination logic 141 can determine one or more distances to the visible tissue and / or the structures of interest 101. One or more outputs from the surface mapping logic 136, tissue identification logic 140, and distance determination logic 141 are provided to the imaging logic 138 and may be combined, integrated, and / or overlaid for communication to a clinician via a display 146 of the imaging system 142.

[0032] 2A-2C, various aspects of the control circuitry 132 for controlling various aspects of the surgical visualization system 100 are described. Turning now to FIG. 2A, a control circuitry 400 configured to control aspects of the surgical visualization system 100 is shown, according to at least one embodiment of the present disclosure. The control circuitry 400 can be configured to implement various processes described herein. The control circuitry 400 can include a microcontroller including one or more processors 402 (e.g., microprocessors, microcontrollers) coupled to at least one memory circuitry 404. The memory circuitry 404 stores machine executable instructions that, when executed by the processor 402, cause the processor 402 to execute machine instructions for implementing various processes described herein. The processor 402 can be any one of a number of single-core or multi-core processors known in the art. The memory circuitry 404 can include volatile and non-volatile storage media. The processor 402 can include an instruction processing unit 406 and an arithmetic unit 408. The instruction processing unit may be configured to receive instructions from a memory circuit 404 of the present disclosure.

[0033] 2B illustrates a combinatorial logic circuit 410 configured to control aspects of the surgical visualization system 100 in accordance with at least one embodiment of the present disclosure. The combinatorial logic circuit 410 can be configured to implement various processes described herein. The combinatorial logic circuit 410 may include a finite state machine including combinatorial logic 412 configured to receive data associated with a surgical instrument or tool at an input 414, process the data through the combinatorial logic 412, and provide an output 416.

[0034] FIG. 2C illustrates a sequential logic circuit 420 configured to control aspects of the surgical visualization system 100 according to at least one embodiment of the present disclosure. The sequential logic circuit 420 or the combinatorial logic 422 can be configured to implement various processes described herein. The sequential logic circuit 420 may include a finite state machine. The sequential logic circuit 420 may include, for example, the combinatorial logic 422, at least one memory circuit 424, and a clock 429. The at least one memory circuit 424 can store a current state of the finite state machine. In certain examples, the sequential logic circuit 420 may be synchronous or asynchronous. The combinatorial logic 422 is configured to receive data associated with a surgical device or system from an input 426, process the data through the combinatorial logic 422, and provide an output 428. In other embodiments, the circuit may include a combination of a processor (e.g., the processor 402 of FIG. 2A) and a finite state machine that implements various processes herein. In other aspects, the finite state machine may include a combination of combinational logic (eg, combinational logic 410 of FIG. 2B) and sequential logic 420.

[0035] Referring again to the surgical visualization system 100 of FIG. 1 , the critical structure 101 may be an anatomical structure of interest. For example, the critical structure 101 may be a ureter, an artery such as the superior mesenteric artery, a vein such as the portal vein, a nerve such as the phrenic nerve, and / or a tumor, among other anatomical structures. In other examples, the critical structure 101 may be a foreign body structure in the anatomical field, such as a surgical device, a surgical fastener, a clip, a clasp, a bougie, a band, and / or a plate. Exemplary critical structures are further described herein and in the aforementioned U.S. patent applications, such as U.S. Patent Application No. 16 / 128,192, entitled “VISUALIZATION OF SURGICAL DEVICES,” filed September 11, 2018, each of which is incorporated herein by reference in its entirety.

[0036] In one embodiment, the critical structure 101 may be embedded in the tissue 103. In other words, the critical structure 101 may be located below the surface 105 of the tissue 103. In such an example, the tissue 103 hides the critical structure 101 from the view of the clinician. The critical structure 101 is also blocked from the view of the imaging device 120 by the tissue 103. The tissue 103 may be, for example, fat, connective tissue, adhesions, and / or organs. In other examples, the critical structure 101 may be partially blocked from the view.

[0037] 1 also illustrates a surgical device 102. The surgical device 102 includes an end effector having opposing jaws extending from a distal end of a shaft of the surgical device 102. The surgical device 102 may be any suitable surgical device, such as, for example, a dissection instrument, a stapler, a grasper, a clip applier, and / or an energy device including a monopolar probe, a bipolar probe, an ablation probe, and / or an ultrasonic end effector. Additionally or alternatively, the surgical device 102 may include another imaging or diagnostic modality, such as, for example, an ultrasound device. In one aspect of the present disclosure, the surgical visualization system 100 may be configured to achieve identification of one or more critical structures 101 and proximity of the surgical device 102 to the critical structures 101.

[0038] The imaging device 120 of the surgical visualization system 100 is configured to detect light of various wavelengths, such as, for example, visible light, spectral light waves (visible or invisible), and structured light patterns (visible or invisible). The imaging device 120 may include multiple lenses, sensors, and / or receivers for detecting different signals. For example, the imaging device 120 may be a hyperspectral, multispectral, or selective spectrum camera, as described further herein. The imaging device 120 may also include a waveform sensor 122 (e.g., a spectral imaging sensor, detector, and / or a three-dimensional camera lens). For example, the imaging device 120 may include a right lens and a left lens used together to generate a three-dimensional image of the surgical site by simultaneously recording two two-dimensional images, render a three-dimensional image of the surgical site, and / or determine one or more distances at the surgical site. Additionally or alternatively, the imaging device 120 may be configured to receive images indicative of the topography of visible tissue, as well as the identification and location of hidden critical structures, as described further herein. For example, as shown in FIG. 1, the field of view of the imager 120 can be overlaid with a pattern of light (structured light) on the tissue surface 105 .

[0039] In one aspect, the surgical visualization system 100 may be incorporated into a robotic system 110. For example, the robotic system 110 may include a first robotic arm 112 and a second robotic arm 114. The robotic arms 112, 114 include a rigid structural member 116 and a joint 118, which may include servo motor control. The first robotic arm 112 is configured to manipulate the surgical device 102, and the second robotic arm 114 is configured to manipulate the imaging device 120. The robotic control unit may be configured to issue control movements to the robotic arms 112, 114 that may act on the surgical device 102 and the imaging device 120, for example.

[0040] The surgical visualization system 100 also includes an emitter 106 configured to emit a pattern of light, such as stripes, grid lines, and / or dots, to enable determination of the topography or condition of the surface 105. For example, the projected light array 130 can be used for three-dimensional scanning and alignment on the surface 105. The projected light array 130 can be emitted from an emitter 106 located, for example, on the surgical device 102, and / or one of the robotic arms 112, 114, and / or the imaging device 120. In one aspect, the projected light array 130 is used to determine the surface 105 of the tissue 103 and / or a shape defined intraoperatively by the movement of the surface 105. The imaging device 120 is configured to detect the projected light array 130 reflected from the surface 105 to determine the topography of the surface 105 and various distances to the surface 105.

[0041] In one aspect, the imaging device 120 may also include an optical waveform emitter 123 configured to emit electromagnetic radiation 124 (NIR photons) that can penetrate the surface 105 of the tissue 103 to reach the structure of interest 101. The imaging device 120 and the optical waveform emitter 123 thereon may be positionable by the robotic arm 114. A corresponding waveform sensor 122 (e.g., an image sensor, a spectrometer, or a vibration sensor) on the imaging device 120 is configured to detect the effect of the electromagnetic radiation received by the waveform sensor 122. The wavelength of the electromagnetic radiation 124 emitted by the optical waveform emitter 123 may be configured to enable identification of a type of anatomical and / or physical structure, such as the structure of interest 101. Identification of the structure of interest 101 may be achieved, for example, by spectral analysis, photoacoustics, and / or ultrasound. In one aspect, the wavelength of the electromagnetic radiation 124 may be tunable. The waveform sensor 122 and the optical waveform emitter 123 may include, for example, a multispectral imaging system and / or a selective spectrum imaging system. In other examples, the waveform sensor 122 and the optical waveform emitter 123 can include, for example, an optoacoustic imaging system. In other examples, the optical waveform emitter 123 can be located on a separate surgical device from the imaging device 120.

[0042] The surgical visualization system 100 may also include a distance sensor system 104 configured to determine one or more distances at the surgical site. In one embodiment, the time-of-flight distance sensor system 104 may be a time-of-flight distance sensor system including an emitter, such as emitter 106, and a receiver 108, and may be disposed on the surgical device 102. In other examples, the time-of-flight emitter may be separate from the structured light emitter. In one general embodiment, the emitter 106 portion of the time-of-flight distance sensor system 104 may include a very small laser source, and the receiver 108 portion of the time-of-flight distance sensor system 104 may include a matching sensor. The time-of-flight distance sensor system 104 may detect the "time of flight," i.e., the time it takes for the laser light emitted by the emitter 106 to bounce back to a sensor portion of the receiver 108. The use of a very narrow light source in the emitter 106 allows the distance sensor system 104 to determine the distance to a surface 105 of the tissue 103 directly in front of the distance sensor system 104. With further reference to FIG. 1, d e is the emitter-tissue distance from the emitter 106 to the surface 105 of the tissue 103, and d t is the device-tissue distance from the distal end of the surgical device 102 to the tissue surface 105. The distance sensor system 104 is used to measure the emitter-tissue distance d e The device-tissue distance d can be determined. t can be obtained from the known position of the emitter 106 on the shaft of the surgical device 102 relative to the distal end of the surgical device 102. In other words, if the distance between the emitter 106 and the distal end of the surgical device 102 is known, then the device-tissue distance d t is the emitter-tissue distance d e In certain examples, the shaft of the surgical device 102 can include one or more articulating joints and can be articulated relative to the emitter 106 and the jaws. The articulating configuration can include, for example, an articulated vertebra-like structure. In certain examples, a three-dimensional camera can be utilized to triangulate one or more distances to the surface 105.

[0043] In various examples, the receiver 108 for the time-of-flight distance sensor system 104 may be mounted on another surgical device instead of the surgical device 102. For example, the receiver 108 may be mounted on a cannula or trocar through which the surgical device 102 extends to reach the surgical site. In yet other examples, the receiver 108 for the time-of-flight distance sensor system 104 may be mounted on another robotic control arm (e.g., the robotic arm 114), on another robotically operated movable arm, and / or on a table or fixture in an operating room (OR). In certain examples, the imaging device 120 includes a time-of-flight receiver 108 that uses a line between the emitter 106 on the surgical device 102 and the imaging device 120 to determine the distance from the emitter 106 to the surface 105 of the tissue 103. For example, the distance d e can be triangulated based on the known positions of the emitter 106 (on the surgical device 102) and receiver 108 (on the imaging device 120) of the time-of-flight distance sensor system 104. The three-dimensional position of the receiver 108 may be known and / or aligned with respect to the intra-operative robot coordinate plane.

[0044] In a particular example, the position of the emitter 106 of the time-of-flight distance sensor system 104 can be controlled by a first robotic arm 112 and the position of the receiver 108 of the time-of-flight distance sensor system 104 can be controlled by a second robotic arm 114. In other examples, the surgical visualization system 100 can be utilized separate from a robotic system. In such cases, the distance sensor system 104 can be independent of the robotic system.

[0045] In certain examples, one or more of the robotic arms 112, 114 may be separate from the main robotic system used in the surgical procedure. At least one of the robotic arms 112, 114 may be positioned and aligned to a particular coordinate system without servo motor control. For example, a closed loop control system and / or multiple sensors for the robotic arm 110 may control and / or align the position of the robotic arms 112, 114 relative to a particular coordinate system. Similarly, the positions of the surgical device 102 and the imaging device 120 may be aligned to a particular coordinate system.

[0046] Further referring to FIG. w is the camera-to-structure-of-interest distance from the optical waveform emitter 123 located on the imaging device 120 to the surface of the structure of interest 101, and d A is the depth of the structure of interest 101 below the surface 105 of the tissue 103 (i.e., the distance between the portion of the surface 105 closest to the surgical device 102 and the structure of interest 101). In various embodiments, the time of flight of the optical waveform emitted from an optical waveform emitter 123 located on the imaging device 120 is determined by the camera-to-structure of interest distance d w The use of spectral imaging in combination with a time-of-flight sensor is further described herein. Further, referring now to FIG. 3, in various embodiments of the present disclosure, the depth d of the critical structure 101 relative to the surface 105 of the tissue 103 can be determined. A is the distance d w , and the known positions (and thus the distance d between) of the emitter 106 on the surgical device 102 and the optical waveform emitter 123 on the imaging device 120. x ) by triangulation, and the distance d e and d A The distance d is the sum of y can be determined.

[0047] Additionally or alternatively, the time of flight from the optical waveform emitter 123 can be configured to determine the distance from the optical waveform emitter 123 to the surface 105 of the tissue 103. For example, a first waveform (or range of waveforms) can be used to determine the camera-to-structure of interest distance dw , and a second waveform (or range of waveforms) can be used to determine the distance to the surface 105 of the tissue 103. In such an example, a different waveform can be used to determine the depth of the critical structure 101 below the surface 105 of the tissue 103.

[0048] Additionally or alternatively, in certain instances, the distance d A can be determined from ultrasound, registered magnetic resonance imaging (MRI) or computed tomography (CT) scans. In yet another example, the distance d A can be determined by spectral imaging because the detection signal received by the imaging device can vary based on the type of material. For example, fat can reduce the detection signal in a first way or by a first amount, while collagen can reduce the detection signal in a different second way or by a second amount.

[0049] 4, surgical visualization system 160 includes an optical waveform emitter 123 and a waveform sensor 122 configured to detect a reflected waveform. The optical waveform emitter 123 is positioned at a distance d from a common device, such as surgical device 162, as described further herein. t and d w In such an example, the distance d from the surface 105 of the tissue 103 to the surface of the critical structure 101 can be determined. A can be determined as follows: d A =d w -d t .

[0050] As disclosed herein, various information regarding visible tissue, embedded vital structures, and surgical devices can be determined by utilizing combination methods incorporating one or more time-of-flight distance sensors, spectral imaging, and / or structured light arrays in combination with an image sensor configured to detect spectral wavelengths and structured light arrays. Additionally, the image sensor can be configured to receive visible light to provide an image of the surgical site to the imaging system. Logic or algorithms are used to discern the information received from the time-of-flight sensor, spectral wavelengths, structured light, and visible light and render a three-dimensional image of the surface tissue and underlying anatomical structures. In various examples, the imaging device 120 can include multiple image sensors.

[0051] Camera-critical structure distance d w can also be detected in one or more alternative ways. In one embodiment, a fluoroscopic visualization technique, such as fluorescent indosciedine green (ICG), can be used to illuminate the structure of interest 201 as shown in Figures 6-8. The camera 220 can include two optical waveform sensors 222, 224 that simultaneously capture left and right images of the structure of interest 201 (Figures 7A and 7B). In such an example, the camera 220 can depict the light of the structure of interest 201 below the surface 205 of the tissue 203, at a distance d w can be determined by a known distance between sensors 222 and 224. In certain examples, the distance can be more accurately determined by utilizing more than one camera or by moving a camera between multiple positions. In certain embodiments, one camera can be controlled by a first robotic arm and a second camera by another robotic arm. In such a robotic system, one camera can be a driven camera, for example, on a driven arm. The driven arm and the camera thereon can be programmed, for example, to track the other camera and maintain a particular distance and / or lens angle.

[0052] In yet another embodiment, the surgical visualization system 100 uses two separate waveform receivers (i.e., cameras / image sensors). w 9, if a critical structure 301 or its contents (e.g., a blood vessel or the contents of a vessel) can emit a signal 302, such as by fluoroscopy, the actual location can be triangulated from two separate cameras 320a, 320b at known locations.

[0053] 10A and 10B, in another embodiment, the surgical visualization system uses a dithering or moving camera 440 to measure the distance d wmay be determined. The camera 440 is robotically controlled such that the three-dimensional coordinates at different positions of the camera 440 are known. In various examples, the camera 440 may pivot at the cannula or patient interface. For example, if the critical structure 401 or its contents (e.g., contents of a blood vessel or vessel) can emit a signal, such as by fluoroscopy, the actual position can be triangulated from the camera 440 moving rapidly between two or more known positions. In FIG. 10A, the camera 440 moves axially along axis A. The camera 440 translates a distance d1 approaching the critical structure 401 along axis A, such as by traversing back and forth on a robotic arm, to a position shown as position 440′. As the camera 440 moves the distance d1 and the size of the image relative to the critical structure 401 changes, the distance to the critical structure 401 can be calculated. For example, an axial translation (distance d1) of 4.28 mm may correspond to an angle θ1 of 6.28 degrees and an angle θ2 of 8.19 degrees. Additionally or alternatively, the camera 440 may rotate or sweep along an arc between different positions. Now referring to FIG. 10B, the camera 440 moves axially along axis A and rotates about axis A by an angle θ3. A pivot point 442 for the rotation of the camera 440 is located at the cannula / patient interface. In FIG. 10B, the camera 440 translates and rotates to a position 440''. As the camera 440 moves and the edge of the image relative to the critical structure 401 changes, the distance to the critical structure 401 can be calculated. In FIG. 10B, the distance d2 may be, for example, 9.01 mm and the angle θ3 may be, for example, 0.9 degrees.

[0054] FIG. 5 illustrates a surgical visualization system 500 that is similar in many respects to the surgical visualization system 100. In various examples, the surgical visualization system 500 may be a further example of the surgical visualization system 100. Like the surgical visualization system 100, the surgical visualization system 500 includes a surgical device 502 and an imaging device 520. The imaging device 520 includes a spectral light emitter 523 that is configured to emit spectral light at multiple wavelengths, for example, to obtain a spectral image of the hidden structure. The imaging device 520 may also include a three-dimensional camera and associated electronic processing circuitry, in various examples. The surgical visualization system 500 is shown utilized intraoperatively to identify and facilitate avoidance of certain critical structures, such as the ureter 501a and blood vessels 501b, within an organ 503 (the uterus in this example) that are not visible from the surface.

[0055] The surgical visualization system 500 measures, via structured light, the emitter-tissue distance d from an emitter 506 on the surgical device 502 to a surface 505 of the uterus 503. e The surgical visualization system 500 is configured to determine the emitter-tissue distance d e Based on this, the device-tissue distance d from the surgical device 502 to the surface 505 of the uterus 503 is t The surgical visualization system 500 is also configured to extrapolate the tissue-ureter distance d from the ureter 501a to the surface 505. A , and the camera-ureter distance d from the imaging device 520 to the ureter 501a w 1, for example, the surgical visualization system 500 may be configured to determine the distance d by, for example, a spectral imaging and time-of-flight sensor. w In various examples, the surgical visualization system 500 can determine the tissue-ureter distance d based on other distance and / or surface mapping logic described herein. A (i.e., depth) can be determined (e.g., triangulated).

[0056] 11, there is shown a schematic diagram of a control system 600 for a surgical visualization system, such as, for example, surgical visualization system 100. Control system 600 is a transformation system that integrates spectral signature tissue identification and structured light tissue localization to identify critical structures, particularly when these structures are obscured by other tissues, such as, for example, fat, connective tissue, blood, and / or other organs. Such techniques may also be useful for detecting tissue variations, such as distinguishing tumors and / or diseased tissue from healthy tissue within an organ.

[0057] The control system 600 is configured to implement a hyperspectral imaging and visualization system in which molecular responses are used to detect and identify anatomical structures within the surgical field of view. The control system 600 includes conversion logic 648 to convert tissue data into information usable by the surgeon. For example, variable reflectance based on wavelength against obscured materials can be utilized to identify critical structures in the anatomy. Furthermore, the control system 600 combines the identified spectral signatures with structured light data in an image. For example, the control system 600 can be used to create a three-dimensional data set for surgical use in a system with enhanced image overlay. The technique can be used both intraoperatively and preoperatively using the additional visual information. In various examples, the control system 600 is configured to provide an alert to the clinician upon the proximity of one or more critical structures. Various algorithms can be employed to guide robotic automated and semi-automated approaches based on the surgical procedure and the proximity to critical structures.

[0058] The projected light array is used to determine tissue shape and motion intraoperatively, or flash lidar may be used for tissue surface mapping.

[0059] The control system 600 is configured to detect structures of interest, provide an image overlay of structures of interest, and measure distances to the surface of visible tissue and distances to buried / covered structures of interest. In other examples, the control system 600 can measure distances to the surface of visible tissue or detect structures of interest and provide an image overlay of structures of interest.

[0060] The control system 600 includes a spectrum control circuit 602. The spectrum control circuit 602 may be a field programmable gate array (FPGA) or another suitable circuit configuration, for example as described herein with respect to Figures 2A-2C. The spectrum control circuit 602 includes a processor 604 that receives a video input signal from a video input processor 606. The processor 604 may be configured to perform hyperspectral processing, and may utilize, for example, C / C++ code. The video input processor 606 accepts a video input terminal for control (metadata) data, such as, for example, shutter time, wavelength, and sensor analysis. The processor 604 is configured to process the video input signal from the video input processor 606 and provide a video output signal to a video output processor 608 that includes, for example, a hyperspectral video output terminal for interface control (metadata) data. The video output processor 608 provides a video output signal to an image overlay controller 610.

[0061] The video input processor 606 is coupled to a patient side camera 612 via a patient isolation circuit 614. As previously mentioned, the camera 612 includes a solid state image sensor 634. The patient isolation circuit may include multiple transformers so that the patient is isolated from other circuits in the system. The camera 612 receives intraoperative images via an optical element 632 and an image sensor 634. The image sensor 634 may include, for example, a CMOS image sensor, or may include any of the image sensor technologies discussed herein in connection with, for example, FIG. 2. In one aspect, the camera 612 outputs images in a 14 bit per pixel signal. It will also be understood that higher or lower pixel resolutions may be utilized without departing from the scope of the present disclosure. The isolated camera output signal 613 is provided to a color RGB fusion circuit 616, which processes the camera output signal 613 using hardware registers 618 and a Nios2 coprocessor 620. The color RGB fusion output signal is provided to the video input processor 606 and the laser pulse control circuit 622.

[0062] Laser pulse control circuitry 622 controls a laser light engine 624. Laser light engine 624 outputs multiple wavelengths (λ1, λ2, λ3...λ4) including near infrared (NIR). n ) light. Laser light engine 624 can operate in multiple modes. In one aspect, laser light engine 624 can operate in, for example, two modes. In a first mode, e.g., a normal operation mode, laser light engine 624 outputs an illumination signal. In a second mode, e.g., an identification mode, laser light engine 624 outputs RGBG light and NIR light. In various examples, laser light engine 624 can operate in a polarization mode.

[0063] Light output 626 from laser light engine 624 illuminates targeted anatomical structures within intraoperative surgical site 627. Laser pulse control circuitry 622 also controls a laser pulse controller 628 for a laser pattern projector 630 that projects a laser light pattern 631, such as a grid or pattern of lines and / or dots, at a predetermined wavelength (λ2) onto the surgical tissue or organs at surgical site 627. Camera 612 receives the output patterned light and reflected light through camera optics 632. Image sensor 634 converts the received light into a digital signal.

[0064] The color RGB fusion circuit 616 also outputs signals to the image overlay controller 610 as well as to a video input module 636 for reading a laser light pattern 631 projected by a laser pattern projector 630 onto the targeted anatomical structure at the surgical site 627. A processing module 638 processes the laser light pattern 631 and outputs a first video output signal 640 representing the distance to visible tissue at the surgical site 627. The data is provided to the image overlay controller 610. The processing module 638 also outputs a second video signal 642 representing a three-dimensional rendering shape of the tissue or organ of the targeted anatomical structure at the surgical site.

[0065] The first and second video output signals 640, 642 contain data representative of the location of the structures of interest on the three-dimensional surface model which is provided to an integration module 643. In combination with data from the video output processor 608 of the spectral control circuit 602, the integration module 643 determines the distance d A (FIG. 1) can be determined (e.g., via a triangulation algorithm 644) and the distance d A may be provided to the image overlay controller 610 via a video output processor 646. Such translation logic may include a translation logic circuit 648, an intermediate video monitor 652, and a camera 624 / laser pattern projector 630 positioned at the surgical site 627.

[0066] Pre-operative data 650 from CT or MRI scans can be used to register or align certain three-dimensional deformable tissues in various instances. Such pre-operative data 650 can be provided to the integration module 643 and ultimately to the image overlay controller 610 so that such information can be overlaid with the image from the camera 612 and provided to the video monitor 652. Registration of pre-operative data is further described herein and in the aforementioned U.S. patent applications, such as U.S. Patent Application No. 16 / 128,195, entitled “INTEGRATION OF IMAGING DATA,” filed September 11, 2018, each of which is incorporated herein by reference in its entirety.

[0067] The video monitor 652 can output the merged / augmented image from the image overlay controller 610. The clinician can select and / or switch between different images on one or more monitors. On a first monitor 652a, the clinician can switch between (A) an image in which a three-dimensional rendering of the visible tissue is shown and (B) an augmented image in which one or more hidden structures of interest are overlaid on the three-dimensional rendering of the visible tissue. On a second monitor 652b, the clinician can switch, for example, between one or more hidden structures of interest and / or distance measurements to the surface of the visible tissue.

[0068] Control system 600 and / or its various control circuits may be incorporated into the various surgical visualization systems disclosed herein.

[0069] 12 illustrates a structured (or patterned) light system 700 in accordance with at least one embodiment of the present disclosure. As described herein, structured light, for example in the form of stripes or lines, can be projected from a light source and / or projector 706 onto a surface 705 of a targeted anatomical structure to determine the shape and contours of the surface 705. For example, a camera 720, which may be similar in many ways to the imaging device 120 (FIG. 1), can be configured to detect the projected light pattern on the surface 705. The manner in which the projected pattern deforms upon impacting the surface 705 allows the vision system to calculate depth and surface information of the targeted anatomical structure.

[0070] In certain instances, invisible (or imperceptible) structured light can be utilized, where the structured light is used without interfering with other computer visual tasks that the projected pattern may disrupt. For example, two exactly opposite repeating patterns of infrared light or very fast frame rate visible light can be utilized to prevent interference. Structured light is further described at en.wikipedia.org / wiki / Structured_light.

[0071] As noted above, the various surgical visualization systems described herein can be utilized to visualize a variety of different types of tissues and / or anatomical structures, including tissues and / or anatomical structures that may be obstructed from being visualized by EMR in the visible portion of the spectrum. In one aspect, the surgical visualization system can utilize a spectral imaging system to visualize different types of tissues based on various combinations of constituent materials. In particular, the spectral imaging system can be configured to detect the presence of various constituent materials in the tissue being visualized based on the absorption coefficients of the tissue across various EMR wavelengths. The spectral imaging system can be further configured to characterize the tissue type of the tissue being visualized based on a particular combination of constituent materials. To illustrate, FIG. 13A is a graph 2300 illustrating how the absorption coefficients of various biomaterials vary across the EMR wavelength spectrum. In the graph 2300, the vertical axis 2303 represents the absorption coefficients of the biomaterials (e.g., cm-1 Graph 2300 further shows a first line 2310 representing the absorption coefficient of water at various EMR wavelengths, a second line 2312 representing the absorption coefficient of protein at various EMR wavelengths, a third line 2314 representing the absorption coefficient of melanin at various EMR wavelengths, a fourth line 2316 representing the absorption coefficient of deoxygenated hemoglobin at various EMR wavelengths, a fifth line 2318 representing the absorption coefficient of oxygenated hemoglobin at various EMR wavelengths, and a sixth line 2319 representing the absorption coefficient of collagen at various EMR wavelengths. Different tissue types have different combinations of constituent materials, and thus tissue types visualized by the surgical visualization system may be identified and differentiated according to the particular combinations of constituent materials detected. Thus, a spectral imaging system may be configured to emit EMR at several different wavelengths, to determine the constituent materials of the tissue based on the detected EMR absorption responses at the different wavelengths, and then to characterize the tissue type based on the particular detected combination of constituent materials.

[0072] An example of the use of spectral imaging techniques to visualize different tissue types and / or anatomical structures is shown in FIG. 13B. In FIG. 13B, a spectral emitter 2320 (e.g., spectral light source 150) is utilized by an imaging system to visualize a surgical site 2325. EMR emitted by the spectral emitter 2320 and reflected from tissues and / or structures at the surgical site 2325 can be received by the image sensor 135 (FIG. 2) to visualize the tissues and / or structures, which may be either visible (e.g., disposed on the surface of the surgical site 2325) or obstructed (e.g., underlying other tissues and / or structures at the surgical site 2325). In this example, the imaging system 142 (FIG. 2) can visualize a tumor 2332, an artery 2334, and various anomalies 2338 (i.e., tissues that do not match known or expected spectral signatures) based on spectral signatures characterized by different absorption properties (e.g., absorption coefficients) of constituent materials for each of the different tissue / structure types. The visualized tissues and structures may be displayed on a display screen associated with or coupled to the imaging system 142, such as the imaging system display 146 (FIG. 2), the primary display 2119 (FIG. 18), the non-sterile display 2109 (FIG. 18), the hub display 2215 (FIG. 19), or the device / instrument display 2237 (FIG. 19).

[0073] Additionally, the imaging system 142 may be configured to adjust or update the displayed surgical site visualization according to the identified tissue and / or structure type. For example, the imaging system 142 may display a margin 2330a associated with the tumor 2332 being visualized on a display screen (e.g., display 146). The margin 2330a may indicate an area or amount of tissue that should be resected to ensure complete removal of the tumor 2332. The control system 133 (FIG. 2) may be configured to control or update dimensions of the margin 2330a based on the tissue and / or structure identified by the imaging system 142. In the illustrated example, the imaging system 142 has identified multiple anomalies 2338 within the FOV. Thus, the control system 133 may adjust the displayed margin 2330a to a first updated margin 2330b having a size sufficient to encompass the anomalies 2338. Additionally, the imaging system 142 has also identified an artery 2334 that overlaps with the initially displayed margin 2330a (as indicated by the highlighted region 2336 of the artery 2334). Accordingly, the control system 133 can adjust the displayed margin 2330a to a second, updated margin 2330c having sufficient dimensions to encompass the relevant portion of the artery 2334.

[0074] Tissues and / or structures may also be imaged or characterized according to their reflectance properties across the EMR wavelength spectrum in addition to or instead of their absorption properties as described above with respect to Figures 13A and 13B. For example, Figures 13C-13E show various graphs of reflectance of different types of tissues or structures across different EMR wavelengths. Figure 13C is a graphical representation 1050 of an exemplary ureter signature for an occluder. Figure 13D is a graphical representation 1052 of an exemplary artery signature for an occluder. Figure 13E is a graphical representation 1054 of an exemplary nerve signature for an occluder. The plots in Figures 13C-13E represent the reflectance as a function of wavelength (nm) of specific structures (ureter, artery, and nerve) versus the corresponding reflectance of fat, lung tissue, and blood at the corresponding wavelengths. It should be understood that these graphs are for illustrative purposes only and that other tissues and / or structures may have corresponding detectable reflectance signatures that allow the tissues and / or structures to be identified and visualized.

[0075] In various examples, selected wavelengths for spectral imaging can be identified and utilized based on expected critical structures and / or obstructions at the surgical site (i.e., "selective spectral" imaging). By utilizing selective spectral imaging, the amount of time required to acquire a spectral image can be minimized such that information can be acquired in real-time or near real-time and utilized intraoperatively. In various examples, the wavelengths can be selected by the clinician or by the control circuitry based on input by the clinician. In certain examples, the wavelengths can be selected based on machine learning and / or big data accessible to the control circuitry, for example, via the cloud.

[0076] The application of spectral imaging to tissue as described above can be utilized intraoperatively to measure distances between a waveform emitter and critical structures that are occluded by tissue. In one aspect of the disclosure, and now referring to Figures 14 and 15, a time-of-flight sensor system 1104 utilizing waveforms 1124, 1125 is shown. The time-of-flight sensor system 1104 can be incorporated into a surgical visualization system 100 (Figure 1) in certain examples. The time-of-flight sensor system 1104 includes a waveform emitter 1106 and a waveform receiver 1108 on the same surgical device 1102. The emitted waves 1124 extend from the emitter 1106 to the critical structure 1101, and the received waves 1125 are reflected by the receiver 1108 from the critical structure 1101. The surgical device 1102 is positioned through a trocar 1110 extending into a cavity 1107 of the patient.

[0077] The waveforms 1124, 1125 are configured to transmit through the obstructing tissue 1103. For example, the wavelengths of the waveforms 1124, 1125 can be wavelengths in the NIR or SWIR spectrum. In one aspect, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or photoacoustic signal can be emitted from the emitter 1106 and can be transmitted through the tissue 1103 obscuring the critical structure 1101. The emitted waveform 1124 can be reflected by the critical structure 1101. The received waveform 1125 can be delayed due to a distance d between the distal end of the surgical device 1102 and the critical structure 1101. In various examples, the waveforms 1124, 1125 can be selected to target the critical structure 1101 in the tissue 1103 based on a spectral signature of the critical structure 1101, as further described herein. In various examples, the emitter 1106 is configured to provide a binary signal, on and off, that can be measured by the receiver 1108, as shown, for example, in FIG.

[0078] Based on the delay between the emitted wave 1124 and the received wave 1125, the time-of-flight sensor system 1104 is configured to determine the distance d (FIG. 14). A time-of-flight timing diagram 1130 for the emitter 1106 and receiver 1108 of FIG. 14 is shown in FIG. 15. The delay is a function of the distance d, which is given by the following equation:

[0079]

number

[0080] As provided herein, the time of flight of the waveforms 1124, 1125 corresponds to the distance d in FIG. 14. In various examples, the additional emitter / receiver and / or the pulsed signal from the emitter 1106 can be configured to emit a non-penetrating signal. The non-penetrating tissue can be configured to determine the distance from the emitter to the surface 1105 of the obstructing tissue 1103. In various examples, the depth of the critical structure 1101 can be determined by the following formula: d A =d w -d t . During the ceremony, d A = Depth of critical structure 1101, d w = distance from emitter 1106 to critical structure 1101 (d in FIG. 14), and d t = the distance from the emitter 1106 (on the distal end of the surgical device 1102) to the surface 1105 of the interfering tissue 1103.

[0081] In one aspect of the disclosure, and referring now to FIG. 16, a time-of-flight sensor system 1204 utilizing waves 1224a, 1224b, 1224c, 1225a, 1225b, 1225c is shown. The time-of-flight sensor system 1204 can be incorporated into the surgical visualization system 100 (FIG. 1) in certain examples. The time-of-flight sensor system 1204 includes a waveform emitter 1206 and a waveform receiver 1208. The waveform emitter 1206 is positioned on a first surgical device 1202a and the waveform receiver 1208 is positioned on a second surgical device 1202b. The surgical devices 1202a, 1202b are positioned through trocars 1210a, 1210b, respectively, that extend into a cavity 1207 of a patient. Emitted waves 1224a, 1224b, 1224c extend from the emitter 1206 towards the surgical site, and received waves 1225a, 1225b, 1225c are reflected to the receiver 1208 from various structures and / or surfaces at the surgical site.

[0082] The different emitted waves 1224a, 1224b, 1224c are configured to target different types of material at the surgical site. For example, the waves 1224a target the obstructing tissue 1203, the waves 1224b target the first critical structure 1201a (e.g., blood vessels), and the waves 1224c target the second critical structure 1201b (e.g., cancerous tumors). The wavelengths of the waves 1224a, 1224b, 1224c may be in the visible, NIR, or SWIR spectrum. For example, the visible light may be reflected by the surface 1205 of the tissue 1203, and the NIR and / or SWIR waveforms may be configured to transmit through the surface 1205 of the tissue 1203. In various aspects, a spectral signal (e.g., hyperspectral, multispectral, or selective spectrum) or a photoacoustic signal may be emitted from the emitter 1206 as described herein. In various examples, the waves 1224b, 1224c can be selected to target critical structures 1201a, 1201b within tissue 1203 based on the spectral signatures of the critical structures 1201a, 1201b, as described further herein. Photoacoustic imaging is further described in various U.S. patent applications which are incorporated herein by reference in this disclosure.

[0083] The emitted waves 1224a, 1224b, 1224c may be reflected from the target material (i.e., the surface 1205, the first structure of interest 1201a, and the second structure 1201b, respectively). The received waveforms 1225a, 1225b, 1225c may be reflected from the target material (i.e., the surface 1205, the first structure of interest 1201a, and the second structure 1201b, respectively) at a distance d 1a , d 2a , d 3a , d 1b , d 2b , d 2c This can cause delays.

[0084] In a time-of-flight sensor system 1204 in which the emitter 1206 and receiver 1208 are independently positionable (e.g., on separate surgical devices 1202a, 1202b and / or controlled by separate robotic arms), various distances d 1a , d 2a , d 3a , d 1b, d 2b , d 2c can be calculated from the known positions of the emitter 1206 and receiver 1208. For example, the positions may be known when the surgical devices 1202a, 1202b are robotically controlled. With knowledge of the positions of the emitter 1206 and receiver 1208, as well as the time it takes for the photon stream to target a particular tissue and the information received by the receiver 1208 of that particular response, the distance d 1a , d 2a , d 3a , d 1b , d 2b , d 2c In one embodiment, the distance to the occluded critical structure 1201 a, 1201 b can be triangulated using the transmitted wavelengths. Since the speed of light is constant for any wavelength of light, visible or invisible, the time-of-flight sensor system 1204 can determine a variety of distances.

[0085] With further reference to FIG. 16, in various examples, the image provided to the clinician may rotate the receiver 1208 such that the center of mass of the target structure in the resulting image remains constant, i.e., in a plane perpendicular to the axis of the selected target structure 1203, 1201a, or 1201b. Such an orientation may quickly communicate one or more relevant distances and / or viewpoints relative to the critical structures. For example, as shown in FIG. 16, the surgical site is displayed from a viewpoint where the critical structure 1201a is perpendicular to the viewing plane (i.e., the blood vessels are pointing into and out of the page). In various examples, such an orientation may be a default setting, but the field of view may be rotated or otherwise adjusted by the clinician. In certain examples, the clinician may switch between different surfaces and / or target structures that define the viewpoint of the surgical site provided by the imaging system.

[0086] In various examples, the receiver 1208 may be mounted on a trocar or cannula, such as trocar 1210b, through which the surgical device 1202b is placed. In other examples, the receiver 1208 may be mounted on a separate robotic arm whose three-dimensional position is known. In various examples, the receiver 1208 may be mounted on a moveable arm separate from the robot controlling the surgical device 1202a, or may be mounted on an operating room (OR) table that can be aligned to the robot coordinate plane during surgery. In such examples, the positions of the emitter 1206 and the receiver 1208 may be aligned to the same coordinate plane so that distances can be triangulated from the output of the time-of-flight sensor system 1204.

[0087] The combination of a time-of-flight sensor system, referred to as TOF-NIRS, and near-infrared spectroscopy (NIRS), which is capable of measuring the time-resolved properties of NIR light with nanosecond resolution, can be found in the article entitled "TIME-OF-FLIGHT NEAR-INFRARED SPECTROSCOPY FOR NONDESTRUCTIVE MEASUREMENT OF INTERNAL QUALITY IN GRAPEFRUIT" in the Journal of the American Society for Horticultural Science, May 2013 vol. 138 no. 3 225-228, which is incorporated herein by reference in its entirety and is accessible at journal.ashspublications.org / content / 138 / 3 / 225.full.

[0088] In various examples, the time-of-flight spectral waveform is configured to determine the depth of the critical structure and / or the proximity of the surgical device to the critical structure. Additionally, various surgical visualization systems disclosed herein include surface mapping logic configured to create a three-dimensional rendering of the surface of the visible tissue. In such examples, even if the visible tissue occludes the critical structure, the clinician can recognize the proximity (or lack of proximity) of the surgical device to the critical structure. In one example, a topography of the surgical site is provided on a monitor by the surface mapping logic. If the critical structure is close to the surface of the tissue, the spectral imaging can communicate the location of the critical structure to the clinician. For example, the spectral imaging can detect structures within 5 mm or 10 mm of the surface. In another example, the spectral imaging can detect structures 10 or 20 mm below the surface of the tissue. Based on known limitations of the spectral imaging system, the system is configured to communicate that the critical structure is out of range if it is simply not detected by the spectral imaging system. Thus, the clinician can continue to move the surgical device and / or manipulate the tissue. When a structure of interest moves within range of the spectral imaging system, the system can identify the structure and therefore communicate that the structure is within range. In such an example, a warning can be provided when the structure is first identified and / or further moved within a predefined proximity zone. In such an example, proximity information (i.e., not in proximity) can be provided to the clinician even if the structure of interest is not identified by the spectral imaging system with known boundaries / range.

[0089] Various surgical visualization systems disclosed herein can be configured to identify the presence and / or proximity of critical structures during surgery and alert the clinician before inadvertent dissection and / or cutting damages the critical structures. In various aspects, the surgical visualization system is configured to identify one or more of the following critical structures, such as the ureter, intestine, rectum, nerves (including the phrenic nerve, recurrent laryngeal nerve [RLN], processus facial nerve, vagus nerve, and their branches), blood vessels (including the pulmonary and lobar arteries and veins, the inferior mesenteric artery [IMA] and their branches, the superior rectal artery, the sigmoid artery, and the left colic artery), the superior mesenteric artery (SMA) and its branches (including the middle colic artery, the right colic artery, and the ileocolic artery), the hepatic artery and its branches, the portal vein and its branches, the splenic artery / vein and its branches, the external and internal iliac vessels (lower abdominal), the short gastric artery, the uterine artery, the median sacral vessels, and lymph nodes. Additionally, the surgical visualization system is configured to indicate the proximity of the surgical device to a critical structure and / or to alert the clinician when the surgical device approaches a critical structure.

[0090] Various aspects of the present disclosure provide intraoperative identification of critical structures (e.g., identification of ureters, nerves, and / or blood vessels) and instrument proximity monitoring. For example, various surgical visualization systems disclosed herein may include spectral imaging and surgical instrument tracking, allowing visualization of critical structures below the surface of the tissue, e.g., 1.0-1.5 cm below the surface of the tissue. In other examples, the surgical visualization system may identify structures less than 1.0 cm or more than 1.5 cm below the surface of the tissue. Even surgical visualization systems that can only identify structures within 0.2 mm of the surface may be useful, e.g., when structures are not naturally visible due to depth. In various aspects, the surgical visualization system may extend the clinician's view, e.g., by virtually displaying critical structures as an overlay of a visible white light image on the surface of the visible tissue. The surgical visualization system may provide real-time three-dimensional spatial tracking of the distal tip of the surgical instrument and may provide a proximity alert when the distal tip of the surgical instrument moves within a range of a critical structure, e.g., within 1.0 cm of the critical structure.

[0091] Various surgical visualization systems disclosed herein can identify when an incision is too close to a critical structure. An incision may be "too close" to a critical structure based on temperature (i.e., too hot near a critical structure that may risk damaging / heating / melting the critical structure) and / or tension (i.e., too much tension near a critical structure that may risk damaging / tearing / pulling the critical structure). Such a surgical visualization system can facilitate perivascular incision, for example, when dissecting perivascular tissue prior to ligation. In various examples, a thermal imaging camera can be utilized to read heat at the surgical site and provide a warning to the clinician based on the detected heat and the distance from the tool to the structure. For example, if the temperature of the tool exceeds a predefined threshold (e.g., 120° F.), a warning can be provided to the clinician at a first distance (e.g., 10 mm), and if the temperature of the tool is below the predefined threshold, a warning can be provided to the clinician at a second distance (e.g., 5 mm). The predefined thresholds and / or warning distances may be default settings and / or clinician programmable. Additionally or alternatively, the proximity warning may be tied to a thermal measurement made by the tool itself, such as, for example, a thermocouple measuring heat within the distal jaws of a monopolar or bipolar dissection instrument or vessel sealer.

[0092] The various surgical visualization systems disclosed herein can provide sufficient sensitivity and specificity to critical structures to allow the clinician to confidently proceed with rapid but safe dissection based on standards of care and / or device safety data. The systems can function intraoperatively in real time during a surgical procedure with minimal, and in various instances, no, ionizing radiation risk to the patient or clinician. Conversely, in fluoroscopy, the patient and clinician can be exposed to ionizing radiation, for example, via an x-ray beam utilized to view anatomical structures in real time.

[0093] The various surgical visualization systems disclosed herein can be configured to detect and identify one or more desired types of structures of interest in the forward path of a surgical device, such as, for example, when the path of the surgical device is robotically controlled. Additionally or alternatively, the surgical visualization system can be configured to detect and identify one or more types of structures of interest in the surrounding area and / or in multiple planes / areas of the surgical device, for example.

[0094] The various surgical visualization systems disclosed herein can be easily operated and / or interpreted. Additionally, various surgical visualization systems can incorporate an "override" feature that allows the clinician to override default settings and / or operations. For example, the clinician can selectively turn off warnings from the surgical visualization system and / or get closer to a critical structure than presented by the surgical visualization system, such as when the risk to a critical structure is lower than the risk of avoiding the area (e.g., when removing cancer around a critical structure, the risk of leaving cancerous tissue behind may be greater than the risk of damaging the critical structure).

[0095] The various surgical visualization systems disclosed herein can be incorporated into surgical systems and / or used during surgical procedures with limited impact on workflow. In other words, implementation of the surgical visualization system may not change the way the surgical procedure is performed. Furthermore, the surgical visualization system can be economical compared to the cost of inadvertent cuts. Data has shown a reduction in inadvertent damage to critical structures, which can facilitate increased reimbursement.

[0096] The various surgical visualization systems disclosed herein can operate in real-time or near real-time and well in advance to allow the clinician to anticipate critical structures. For example, the surgical visualization system can provide sufficient time for "slow down, evaluate, and avoid" to maximize the efficiency of the surgical procedure.

[0097] Various surgical visualization systems disclosed herein may not require contrast agents or dyes to be injected into tissue. For example, spectral imaging is configured to visualize hidden structures intraoperatively without the use of contrast agents or dyes. In other examples, contrast agents may be easier to inject into the appropriate layers of tissue than with other visualization systems. The time between injection of contrast agents and visualization of critical structures may be less than two hours, for example.

[0098] The various surgical visualization systems disclosed herein can be coupled with clinical data and / or device data. For example, the data can provide a boundary of the distance of an energy-enabled surgical device (or other potentially damaging device) from tissue that the surgeon does not want to damage. Any data modules that interface with the surgical visualization systems disclosed herein can be provided integrally or separately from the robot, for example, to enable use with standalone surgical devices in open or laparoscopic procedures. The surgical visualization system can be compatible with a robotic surgical system in various examples. For example, visualization images / information can be displayed within the robot console.

[0099] In various instances, the clinician may not know the location of the critical structure relative to the surgical tool. For example, if the critical structure is embedded in tissue, the clinician may be unable to ascertain the location of the critical structure. In certain instances, the clinician may wish to keep the surgical device out of range of locations surrounding the critical structure and / or away from visible tissue that covers the hidden critical structure. If the location of the hidden critical structure is unknown, the clinician may risk getting too close to the critical structure, which may result in inadvertent trauma and / or dissection of the critical structure and / or excessive energy, heat, and / or tension near the critical structure. Alternatively, the clinician may remain far away from suspected critical structures in an effort to avoid the critical structure, risking affecting tissue in undesirable locations.

[0100] A surgical visualization system is provided that provides tracking of a surgical device relative to one or more critical structures. For example, the surgical visualization system can track the proximity of the surgical device to critical structures. Such tracking can occur intraoperatively, in real-time, and / or near real-time. In various examples, the tracking data can be provided to a clinician via a display screen (e.g., a monitor) of the imaging system.

[0101] In one aspect of the disclosure, a surgical visualization system includes a surgical device including an emitter configured to emit a structured light pattern onto a visible surface, an imaging system including a camera configured to detect the embedded structure and the structured light pattern on the visible surface, and a control circuit in signal communication with the camera and the imaging system, the control circuit configured to determine a distance from the surgical device to the embedded structure and provide a signal indicative of the distance to the imaging system. For example, the distance can be determined based on a three-dimensional image of the illuminated structure provided by images from multiple lenses (e.g., left and right lenses) of the camera by calculating the distance from the camera to the illuminated critical structure by fluoroscopy. The distance from the surgical device to the critical structure can be triangulated, for example, based on known positions of the surgical device and the camera. Alternative means for determining the distance to the embedded critical structure are further described herein. For example, a NIR time-of-flight distance sensor can be used. Additionally or alternatively, the surgical visualization system can determine the distance to visible tissue that overlaps / covers the embedded critical structure. For example, the surgical visualization system can identify hidden structures of interest and enhance the image of the hidden structures of interest by drawing an outline of the hidden structures of interest over the visible structures, e.g., a line on the surface of the visible tissue, and the surgical visualization system can further determine the distance to the enhanced line on the visible tissue.

[0102] By providing the clinician with up-to-date information regarding the proximity of the surgical device to hidden critical structures and / or visible structures, as provided by the various surgical visualization systems disclosed herein, the clinician can make more informed decisions regarding the placement of the surgical device relative to the hidden critical structures. For example, the clinician can see the distance between the surgical device and the critical structures in real-time / intra-operatively, and in certain instances, alerts and / or warnings can be provided by the imaging system when the surgical device moves within a predefined proximity and / or zone of the critical structures. In certain instances, alerts and / or warnings can be provided when the trajectory of the surgical device indicates a possibility of colliding with a "no-fly" zone near the critical structure (e.g., within 1 mm, 2 mm, 5 mm, 10 mm, 20 mm or more of the critical structure). In such instances, the clinician can maintain operation throughout the surgical procedure without the need for a certified physician to monitor suspected critical structures and the proximity of the surgical device thereto. As a result, certain surgical procedures can be performed more quickly, with fewer pauses / interruptions, and / or with improved precision and / or certainty. In one aspect, a surgical visualization system can be utilized to detect tissue diversity, for example, tissue diversity within an organ, in order to distinguish tumor / cancerous / pathological tissue from healthy tissue. Such a surgical visualization system can maximize the removal of pathological tissue while minimizing the removal of healthy tissue.

[0103] Surgical Hub System The various visualization or imaging systems described herein may be incorporated into a surgical hub system, such as the one illustrated in connection with Figures 17-19 and described in further detail below.

[0104] 17, a computer implemented interactive surgical system 2100 includes one or more surgical systems 2102 and a cloud-based system (e.g., a cloud 2104 that may include a remote server 2113 coupled to a storage device 2105). Each surgical system 2102 includes at least one surgical hub 2106 in communication with the cloud 2104 that may include the remote server 2113. In one example, as shown in FIG. 17, the surgical system 2102 includes a visualization system 2108, a robotic system 2110, and a handheld intelligent surgical instrument 2112 configured to communicate with each other and / or with the hub 2106. In some aspects, the surgical system 2102 may include M hubs 2106, N visualization systems 2108, O robotic systems 2110, and P handheld intelligent surgical instruments 2112, where M, N, O, and P are integers equal to or greater than 1.

[0105] FIG. 18 shows an example of a surgical system 2102 used to perform a surgical procedure on a patient lying on an operating table 2114 in a surgical room 2116. A robotic system 2110 is used as part of the surgical system 2102 in the surgical procedure. The robotic system 2110 includes a surgeon's console 2118, a patient side cart 2120 (surgical robot), and a surgical robot hub 2122. The patient side cart 2120 can manipulate at least one detachably coupled surgical tool 2117 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 2118. Images of the surgical site can be obtained by a medical imaging device 2124, which can be manipulated and orientated by the patient side cart 2120. The robotic hub 2122 can be used to process images of the surgical site and then display the processed images to the surgeon through the surgeon's console 2118.

[0106] Other types of robotic systems can be readily adapted for use with the surgical system 2102. Various examples of robotic systems and surgical tools suitable for use in the present disclosure are described in various U.S. patent applications which are incorporated herein by reference in this disclosure.

[0107] Various examples of cloud-based analytics performed by the cloud 2104 and suitable for use in the present disclosure are described in various US patent applications which are incorporated herein by reference in this disclosure.

[0108] In various aspects, the imager 2124 includes at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, Charge-Coupled Device (CCD) sensors and Complementary Metal-Oxide Semiconductor (CMOS) sensors.

[0109] The optical components of the imager 2124 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate a portion of the surgical field. The one or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0110] The one or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectrum. The visible spectrum, sometimes referred to as the optical spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to (i.e., detectable by) the human eye, and may also be referred to as visible light, or simply light. A typical human eye responds to wavelengths in air between about 380 nm and about 750 nm.

[0111] The invisible spectrum (i.e., the non-radiative spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths less than about 380 nm and greater than about 750 nm). The invisible spectrum is not detectable by the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum, which are invisible infrared (IR), microwave, and wireless electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum, which are invisible ultraviolet, X-ray, and gamma ray electromagnetic radiation.

[0112] In various aspects, the imaging device 2124 is configured for use in minimally invasive surgery. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cystoscopes, duodenoscopes, enteroscopes, esophagogastroduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0113] In one aspect, the imaging device uses multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from across the electromagnetic spectrum. The wavelengths can be separated by filters or by using instruments that can sense specific wavelengths of light, including frequencies beyond the visible light range, e.g., IR and ultraviolet light. Spectral imaging can allow for the extraction of additional information that the human eye cannot capture with its red, green and blue receptors. The use of multispectral imaging is described in various U.S. patent applications, which are incorporated herein by reference in this disclosure. Multispectral monitoring can be a useful tool to reposition the surgical field after a surgical task is completed to perform one or more of the above-mentioned tests on the tissue being treated.

[0114] It is self-evident that any surgical procedure requires strict sterilization of the operating room and surgical equipment. The strict sanitary and sterile conditions required in the "surgical theater", i.e., the operating room or treatment room, require the utmost sterility of all medical devices and equipment. Part of the sterilization process described above includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 2124 and its accessories and components. It is understood that the sterile field may be considered a specific area that is deemed free of microorganisms, such as in a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is prepared for the surgical procedure. The sterile field may include cleaned team members in appropriate clothing, and all the fixtures and fixtures in the area.

[0115] In various aspects, the visualization system 2108 includes one or more imaging sensors strategically positioned relative to the sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 18. In one aspect, the visualization system 2108 includes HL7, PACS, and EMR interfaces. Various components of the visualization system 2108 are described in various U.S. patent applications which are incorporated herein by reference in this disclosure.

[0116] As shown in FIG. 18 , the primary display 2119 is positioned within the sterile field so as to be visible to an operator positioned at the operating table 2114. Additionally, a visualization tower 21121 is positioned outside the sterile field. The visualization tower 21121 includes a first non-sterile display 2107 and a second non-sterile display 2109 facing away from each other. A visualization system 2108 guided by the hub 2106 is configured to utilize the displays 2107, 2109, and 2119 to coordinate information flow to operators inside and outside the sterile field. For example, the hub 2106 can cause the visualization system 2108 to maintain a live view of the surgical site on the primary display 2119 while displaying snapshots of the surgical site recorded by the imager 2124 on the non-sterile displays 2107 or 2109. The snapshots on the non-sterile displays 2107 or 2109 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0117] In one aspect, the hub 2106 is also configured to send diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 to a primary display 2119 in the sterile field for viewing by a sterile operator located at the operating table. In one example, the input may be in the form of a correction to a snapshot displayed on the non-sterile display 2107 or 2109 that can be sent by the hub 2106 to the primary display 2119.

[0118] 18, a surgical instrument 2112 is used as part of the surgical system 2102 in a surgical procedure. The hub 2106 is also configured to coordinate the flow of information to the display of the surgical instrument 2112, as described in various U.S. patent applications incorporated herein by reference in this disclosure. Diagnostic input or feedback entered by a non-sterile operator at the visualization tower 21121 can be sent by the hub 2106 to the surgical instrument display 2115 in the sterile field, where the diagnostic input or feedback can be viewed by the operator of the surgical instrument 2112. Exemplary surgical instruments suitable for use in the surgical system 2102 are described in various U.S. patent applications incorporated herein by reference in this disclosure.

[0119] 19 illustrates a computer implemented interactive surgical system 2200. The computer implemented interactive surgical system 2200 is similar in many respects to the computer implemented interactive surgical system 2100. The surgical system 2200 includes at least one surgical hub 2236 in communication with a cloud 2204, which may include a remote server 2213. In one aspect, the computer implemented interactive surgical system 2200 includes a surgical hub 2236 connected to a plurality of operating room devices, such as, for example, intelligent surgical instruments, robots, and other computerized devices located in the operating room. The surgical hub 2236 includes a communication interface for communicatively coupling the surgical hub 2236 to the cloud 2204 and / or the remote server 2213. As shown in the embodiment of FIG. 19, the surgical hub 2236 is coupled to an imaging module 2238 coupled to an endoscope 2239, a generator module 2240 coupled to an energy device 2421, a smoke evacuator module 2226, a suction / irrigation module 2228, a communication module 2230, a processor module 2232, a storage array 2234, a smart device / instrument 2235, optionally coupled to a display 2237, and a non-contact sensor module 2242. The operating room devices are coupled to cloud computing resources and data storage via the surgical hub 2236. The robotic hub 2222 may also be connected to the surgical hub 2236 and cloud computing resources. The devices / instruments 2235, visualization system 2209, among others, may be coupled to the surgical hub 2236 via wired or wireless communication standards or protocols as described herein. The surgical hub 2236 may be coupled to a hub display 2215 (e.g., monitor, screen) for displaying and overlaying images received from the imaging modules, device / instrument displays and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower along with images and overlay images.

[0120] Situational Awareness Various visualization systems or aspects of visualization systems described herein can be utilized as part of a situational awareness system that may be embodied or executed by the surgical hubs 2106, 2236 (FIGS. 17-19). In particular, characterizing, identifying, and / or visualizing surgical instruments or other surgical devices (including their positions, orientations, and movements), tissues, structures, users, and others located in the surgical field or room can provide contextual data that can be utilized by the situational awareness system to infer the type of surgical procedure or steps thereof to be performed, the type of tissue and / or structure to be manipulated by the surgeon, etc. This contextual data can then be utilized by the situational awareness system to provide alerts to the user, suggest subsequent steps or actions for the user to perform, prepare the surgical device in anticipation of the user's use (e.g., activate an electrosurgical generator in anticipation of an electrosurgical instrument being utilized in a subsequent step of the surgical procedure), intelligently control the surgical instrument (e.g., customize the operating parameters of the surgical instrument based on the particular health profile of each patient), etc.

[0121] Although an "intelligent" device that includes a control algorithm responsive to sensed data may be an improvement over a "dumb" device that operates without considering the sensed data, some sensed data may be incomplete or inconclusive when considered alone, i.e., without the context of the type of surgical procedure being performed or the type of tissue being operated on. Without knowing the procedure context (e.g., knowing the type of tissue being operated on or the type of procedure being performed), the control algorithm may control the modular device inaccurately or suboptimally when given sensed data that does not include the particular context. The modular device may include any surgical device that can be controlled by a situational awareness system, such as a visualization system device (e.g., a camera or display screen), surgical instruments (e.g., ultrasonic surgical instruments, electrosurgical instruments, or surgical staplers), and other surgical devices (e.g., smoke evacuation devices). For example, the optimal method for a control algorithm to control a surgical instrument in response to a particular sensed parameter may vary depending on the particular type of tissue being operated on. This is due to the fact that different tissue types have different properties (e.g., resistance to tearing) and therefore respond differently to actions taken by a surgical instrument. Thus, it may be desirable for the surgical instrument to take different actions even when the same measurements are sensed for a particular parameter. As one specific example, the optimal way to control a surgical stapling and severing instrument in response to sensing an unexpectedly high force to close its end effector depends on whether the tissue type is susceptible to tearing or resistant to this. For tissue that is susceptible to tearing, such as lung tissue, the instrument's control algorithm optimally ramps down the motor in response to an unexpectedly high force to close to avoid tearing the tissue. For tissue that is resistant to tearing, such as stomach tissue, the instrument's control algorithm optimally ramps up the motor in response to an unexpectedly high force to close to ensure that the end effector is properly clamped to the tissue.Without knowing whether lung or stomach tissue is being clamped, the control algorithm may make suboptimal decisions.

[0122] One solution utilizes a surgical hub that includes a system configured to derive information about the surgical procedure being performed based on data received from various data sources and then control the paired modular devices accordingly. In other words, the surgical hub is configured to infer information about the surgical procedure from the received data and then control the modular devices paired with the surgical hub based on the inferred context of the surgical procedure. FIG. 20 illustrates a diagram of a situation-aware surgical system 2400 according to at least one aspect of the present disclosure. In some examples, the data sources 2426 include, for example, a modular device 2402 (which may include sensors configured to detect parameters associated with the patient and / or the modular device itself), a database 2422 (e.g., an EMR database including patient records), and a patient monitoring device 2424 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor).

[0123] The surgical hub 2404 may be similar in many respects to the hub 106 and may be configured to derive contextual information related to a surgical procedure from the data received, for example, based on a particular combination of data or a particular order in which data is received from the data sources 2426. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the particular step of the surgical procedure the surgeon is performing, the type of tissue being operated on, or the body cavity that is the subject of the procedure. This ability by some aspects of the surgical hub 2404 to derive or infer information related to a surgical procedure from the received data may be referred to as "situational awareness." In one example, the surgical hub 2404 may incorporate a situational awareness system, which is hardware and / or programming associated with the surgical hub 2404 that derives contextual information related to a surgical procedure from the received data.

[0124] The situational awareness system of the surgical hub 2404 may be configured to derive contextual information from data received from the data sources 2426 in a variety of different ways. In one example, the situational awareness system includes a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained with training data to correlate various inputs (e.g., data from the database 2422, the patient monitoring devices 2424, and / or the modular devices 2402) with corresponding contextual information for the surgical procedure. In other words, the machine learning system may be trained to accurately derive contextual information for the surgical procedure from the inputs provided. In another example, the situational awareness system may include a lookup table that stores pre-characterized contextual information for the surgical procedure in correspondence with one or more inputs (or ranges of inputs) that correspond to the contextual information. In response to being queried by one or more inputs, the lookup table may return corresponding contextual information for the situational awareness system to control the modular devices 2402. In one example, the contextual information received by the surgical hub 2404 situational awareness system is associated with a particular control adjustment or set of control adjustments for one or more modular devices 2402. In another example, the situational awareness system includes an additional machine learning system, lookup table, or other such system that generates or looks up one or more control adjustments for one or more modular devices 2402 when provided with the contextual information as an input.

[0125] A surgical hub 2404 incorporating a situational awareness system provides many benefits to the surgical system 2400. One benefit includes improved interpretation of sensed and collected data, which in turn improves processing accuracy and / or use of the data during the course of a surgical procedure. To return to the previous example, the situationally aware surgical hub 2404 can determine what type of tissue is being operated on, and thus, when an unexpectedly high force is detected to close the end effector of the surgical instrument, the situationally aware surgical hub 2404 can properly ramp up or down the motor of the surgical instrument to match the type of tissue.

[0126] As another example, the type of tissue being operated on may affect adjustments made to the compression speed and load threshold of the surgical stapling and severing instrument for a particular tissue gap measurement. The context-aware surgical hub 2404 may estimate whether the surgical procedure being performed is a thoracic or abdominal procedure, which allows the surgical hub 2404 to determine whether the tissue being clamped by the end effector of the surgical stapling and severing instrument is pulmonary (in the case of thoracic surgery) or stomach (in the case of abdominal surgery). The surgical hub 2404 may then adjust the compression speed and load threshold of the surgical stapling and severing instrument appropriately for the type of tissue.

[0127] As yet another example, the type of body cavity being operated on during an insufflation procedure may affect the functionality of the smoke evacuator. The context-aware surgical hub 2404 can determine if the surgical site is under pressure (by determining that a surgical procedure is utilizing insufflation) and determine the procedure type. Since the procedure type is generally performed within a particular body cavity, the surgical hub 2404 can control the smoke evacuator motor speed appropriately for the body cavity being operated on. Thus, the context-aware surgical hub 2404 can provide a consistent amount of smoke evacuation for both thoracic and abdominal procedures.

[0128] As yet another example, the type of procedure being performed may affect the optimal energy level at which an ultrasonic surgical instrument or a radio frequency (RF) electrosurgical instrument operates. For example, an arthroscopic procedure requires a higher energy level because the end effector of the ultrasonic surgical instrument or RF electrosurgical instrument is submerged in fluid. The context-aware surgical hub 2404 may determine whether the surgical procedure is an arthroscopic procedure. The surgical hub 2404 may then adjust the RF power level or ultrasonic amplitude (i.e., "energy level") of the generator to compensate for the fluid-filled environment. Relatedly, the type of tissue being operated on may affect the optimal energy level at which an ultrasonic surgical instrument or a RF electrosurgical instrument operates. The context-aware surgical hub 2404 may determine what type of surgical procedure is being performed and then customize the energy level of the ultrasonic surgical instrument or the RF electrosurgical instrument, respectively, according to the expected tissue profile for the surgical procedure. Additionally, the context-aware surgical hub 2404 may be configured to adjust the energy levels of the ultrasonic surgical instrument or RF electrosurgical instrument throughout the course of a surgical procedure, rather than just by procedure. The context-aware surgical hub 2404 can determine which step of the surgical procedure is being performed or will be performed subsequently, and then update the generator and / or the control algorithms of the ultrasonic surgical instrument or RF electrosurgical instrument to set the energy levels to a value appropriate for the tissue type expected according to the step of the surgical procedure.

[0129] As yet another example, the surgical hub 2404 may draw data from additional data sources 2426 to improve conclusions the surgical hub 2404 draws from one data source 2426. The context-aware surgical hub 2404 may augment data received from the modular device 2402 with contextual information built about the surgical procedure from other data sources 2426. For example, the context-aware surgical hub 2404 may be configured to determine whether hemostasis has occurred (i.e., whether bleeding at the surgical site has stopped) according to video or image data received from a medical imaging device. However, in some cases, the video or image data may not be deterministic. Thus, in one example, the surgical hub 2404 may be further configured to compare a physiological measurement (e.g., blood pressure sensed by a BP monitor communicatively connected to the surgical hub 2404) with visual or image data of hemostasis (e.g., from a medical imaging device 124 (FIG. 2) communicatively coupled to the surgical hub 2404) to make a determination about the integrity of a staple line or tissue weld. In other words, the situational awareness system of the surgical hub 2404 can take into account the physiological measurement data to provide additional context when analyzing the visualization data, which can be useful in cases where the visualization data may not be definitive or incomplete by itself.

[0130] Another benefit includes proactively and automatically controlling the paired modular devices 2402 according to the particular step of the surgical procedure being performed to reduce the number of times a medical professional is required to interact with or control the surgical system 2400 during the course of the surgical procedure. For example, the situation-aware surgical hub 2404 may proactively activate the generator to which the RF electrosurgical instrument is connected if it determines that a subsequent step of the procedure requires the use of the instrument. By proactively activating the energy source, the instrument may be ready for use as soon as the preceding step of the procedure is completed.

[0131] As another example, the context-aware surgical hub 2404 can determine whether a current or subsequent step in a surgical procedure requires a different view or degree of magnification on the display according to features at the surgical site that the surgeon is expected to need to see. The surgical hub 2404 can then proactively modify the displayed view (e.g., provided by a medical imaging device for the visualization system 108) appropriately, so that the display automatically adjusts throughout the surgical procedure.

[0132] As yet another example, the context-aware surgical hub 2404 can determine which step of a surgical procedure is being performed or will be performed later, and whether specific data or comparisons between data are required for that step of the surgical procedure. The surgical hub 2404 can be configured to automatically call up data screens based on the step of the surgical procedure being performed, without waiting for the surgeon to ask for specific information.

[0133] Another benefit includes checking for errors during the setup or course of a surgical procedure. For example, the context-aware surgical hub 2404 can determine whether the surgical site is properly or optimally set up for the surgical procedure being performed. The surgical hub 2404 can be configured to determine the type of surgical procedure being performed, retrieve (e.g., from memory) the corresponding checklist, product location, or setup needs, and then compare the current surgical site layout to a standard layout for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 can be configured to compare, for example, a list of items for the procedure scanned by a preferred scanner and / or a list of devices paired with the surgical hub 2404 to a recommended or expected manifest of items and / or devices for a given surgical procedure. If a discontinuity exists between the lists, the surgical hub 2404 can be configured to provide a warning indicating that a particular modular device 2402, patient monitoring device 2424, and / or other surgical item is missing. In one example, the surgical hub 2404 may be configured to determine the relative distance or position of the modular devices 2402 and the patient monitoring devices 2424, for example, by proximity sensors. The surgical hub 2404 may compare the relative positions of the devices to a recommended or expected layout for a particular surgical procedure. If a discontinuity exists between the layouts, the surgical hub 2404 may be configured to provide a warning indicating that the current layout of the surgical procedure deviates from the recommended layout.

[0134] As another example, the context-aware surgical hub 2404 may determine whether a surgeon (or other medical personnel) is making an error or otherwise deviating from a desired course of action during the course of a surgical procedure. For example, the surgical hub 2404 may be configured to determine the type of surgical procedure being performed, retrieve (e.g., from a memory) a corresponding list of steps or sequences of equipment usage, and then compare the steps being performed, or equipment being used, during the course of the surgical procedure with the expected steps or equipment for the type of surgical procedure that the surgical hub 2404 has determined is being performed. In one example, the surgical hub 2404 may be configured to provide a warning indicating that an unexpected action is being performed or an unexpected device is being utilized at a particular step in the surgical procedure.

[0135] Overall, the situational awareness system for the surgical hub 2404 improves the outcome of surgical procedures by adjusting surgical instruments (and other modular devices 2402) for the specific context of each surgical procedure (e.g., adjusting for different tissue types) and validating actions during the surgical procedure. The situational awareness system also improves the efficiency of the surgeon in performing a surgical procedure by automatically suggesting next steps, providing data, and adjusting displays and other modular devices 2402 within the surgical site according to the specific context of the procedure.

[0136] 21, there is shown a timeline 2500 illustrating the situational awareness of a hub, such as the surgical hub 106 or 206 (FIGS. 1-11). The timeline 2500 illustrates an exemplary surgical procedure and the contextual information that the surgical hub 106, 206 may derive from data received from data sources at each step of the surgical procedure. The timeline 2500 illustrates typical steps that nurses, surgeons, and other medical personnel may take during the course of a lung segmentectomy procedure, beginning with setting up the operating room and ending with transporting the patient to a post-operative recovery room.

[0137] The situation-aware surgical hub 106, 206 receives data from data sources throughout the course of a surgical procedure, including data generated each time medical personnel use a modular device paired with the surgical hub 106, 206. The surgical hub 106, 206 receives this data from the paired modular devices and other data sources, and can continually derive inferences (i.e., contextual information) regarding the ongoing procedure as new data is received, such as which step of the procedure is being performed at any given time. The situational awareness system of the surgical hub 106, 206 can, for example, record data regarding the procedure to generate a report, verify steps being taken by medical personnel, provide data or prompts (e.g., via a display screen) that may be relevant to a particular procedure step, adjust the modular device based on the context (e.g., activate a monitor, adjust the field of view (FOV) of a medical imaging device, or change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument, etc.), and any other such actions described above.

[0138] As a first step 2502 in this exemplary procedure, hospital personnel retrieve the patient's EMR from the hospital's EMR database. Based on selected patient data in the EMR, the surgical hub 106, 206 determines that the procedure to be performed is a thoracic procedure.

[0139] In a second step 2504, the personnel scans the incoming medical supplies for the procedure. The surgical hub 106, 206 cross-references the scanned supplies with a list of supplies utilized in various types of procedures to verify that the combination of supplies is compatible with a thoracic procedure. Additionally, the surgical hub 106, 206 may also determine that the procedure is not a wedge procedure (either because the incoming supplies do not include the specific supplies required for a thoracic wedge procedure or are not otherwise compatible with a thoracic wedge procedure).

[0140] In a third step 2506, medical personnel scan the patient's band via a scanner communicatively connected to the surgical hub 106, 206. The surgical hub 106, 206 can then verify the patient's identity based on the scanned data.

[0141] In a fourth step 2508, the medical staff turns on the auxiliary devices. The auxiliary devices utilized may vary according to the type of surgical procedure and the technology used by the surgeon, but in this exemplary case, they include smoke evacuators, aspirators, and medical imaging devices. When the auxiliary device is activated, the auxiliary device, which is a modular device, may automatically pair with the surgical hub 106, 206 located within a certain proximity of the modular device as part of its initialization process. The surgical hub 106, 206 may then derive contextual information regarding the surgical procedure by detecting the type of modular device that is paired with it during this pre-operative or initialization phase. In this particular example, the surgical hub 106, 206 determines that the surgical procedure is a VATS procedure based on this particular combination of paired modular devices. Based on the data from the patient's EMR, the list of medical supplies used in the procedure, and the combination of the types of modular devices connecting to the hub, the surgical hub 106, 206 may roughly deduce the particular procedure that the surgical team will be performing. Once the surgical hub 106, 206 knows what particular procedure is being performed, it can then retrieve the steps of that procedure from memory or from the cloud and then cross-reference data subsequently received from connected data sources (e.g., modular devices and patient monitoring devices) to estimate which steps of the surgical procedure the surgical team is performing.

[0142] In a fifth step 2510, personnel attach EKG electrodes and other patient monitoring devices to the patient. The EKG electrodes and other patient monitoring devices may be paired with the surgical hub 106, 206. Once the surgical hub 106, 206 begins receiving data from the patient monitoring devices, the surgical hub 106, 206 confirms that the patient is in the operating room.

[0143] In a sixth step 2512, medical personnel anesthetize the patient. The surgical hub 106, 206 may estimate that the patient is under anesthesia based on data from the modular devices and / or patient monitoring devices, including, for example, EKG data, blood pressure data, ventilator data, or a combination thereof. Once the sixth step 2512 is completed, the pre-operative portion of the lung segmentectomy procedure is complete and the operative portion begins.

[0144] In a seventh step 2514, the patient's lung being operated on is collapsed (while ventilation is switched to the contralateral lung). The surgical hub 106, 206 can, for example, infer that the patient's lung has been collapsed from ventilator data. Because the surgical hub 106, 206 can compare the detection of the patient's lung being collapsed to the expected steps of the procedure (which may be accessed or read in advance), it can infer that the surgical portion of the procedure has already begun, thereby determining that collapsing the lung is the first surgical step in this particular procedure.

[0145] In an eighth step 2516, a medical imaging device (e.g., a scope) is inserted and video feed from the medical imaging device is initiated. The surgical hub 106, 206 receives the medical imaging device data (i.e., video or image data) through a connection to the medical imaging device. Upon receiving the medical imaging device data, the surgical hub 106, 206 can determine that the laparoscopic portion of the surgical procedure has begun. Additionally, the surgical hub 106, 206 can determine that the particular procedure being performed is a segmentectomy as opposed to a lobectomy (note that a wedge procedure has already been determined not to be a possibility by the surgical hub 106, 206 based on the data received in the second step 2504 of the procedure). Data from the medical imaging device 124 (FIG. 2) may be used in many different ways to determine contextual information regarding the type of procedure being performed. For example, it may be used to determine the angle at which the medical imaging device is oriented relative to the visualization of the patient's anatomy, monitor the number of medical imaging devices being used (i.e., activated and paired with the surgical hub 106, 206), and monitor the type of visualization device being used. For example, one technique for performing a VATS lobectomy places the camera above the diaphragm in the anterior inferior corner of the patient's thoracic cavity, while one technique for performing a VATS segmentectomy places the camera in an intercostal position anterior to the segmental fissure. For example, using pattern recognition or machine learning techniques, the situational awareness system may be trained to recognize the location of the medical imaging device based on the visualization of the patient's anatomy. As another example, one technique for performing a VATS lobectomy utilizes a single medical imaging device, while another technique for performing a VATS segmentectomy utilizes multiple cameras. As yet another example, one technique for performing a VATS segmentectomy uses an infrared light source (which may be communicatively coupled to the surgical hub as part of the visualization system) to visualize the segmental fissure, which is not used in a VATS lobectomy.By tracking any or all of this data from the medical imaging devices, the surgical hub 106, 206 can determine the particular type of surgical procedure being performed and / or the technique being used for the particular type of surgical procedure.

[0146] In a ninth step 2518, the surgical team begins the incision step of the procedure. Because the surgical hub 106, 206 receives data from the RF or ultrasonic generator indicating that an energy instrument is being fired, it can presume that the surgeon is in the process of incising and mobilizing the patient's lungs. The surgical hub 106, 206 can cross-reference the received data with the retrieved step of the surgical procedure to determine that the energy instrument being fired at this point in the process (i.e., after the steps of the procedure described above have been completed) corresponds to the incision step. In a particular example, the energy instrument can be an energy tool mounted on a robotic arm of a robotic surgical system.

[0147] In a tenth step 2520, the surgical team proceeds to the ligation step of the procedure. Because the surgical hub 106, 206 receives data from the surgical stapling and severing instrument indicating that the instrument is being fired, it can infer that the surgeon is currently ligating the artery and vein. As in the previous step, the surgical hub 106, 206 can derive this inference by cross-referencing the receipt of data from the surgical stapling and severing instrument with the step in the retrieved process. In a particular example, the surgical instrument may be a surgical tool mounted on a robotic arm of a robotic surgical system.

[0148] In an eleventh step 2522, the segmental resection portion of the procedure is performed. The surgical hub 106, 206 can infer that the surgeon is transecting parenchymal tissue based on data from the surgical stapling and cutting instrument, including data from its cartridge. The cartridge data can correspond, for example, to the size or type of staples fired by the instrument. Because different types of staples are utilized for different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transected. In this case, the type of staples fired is utilized for parenchymal tissue (or other similar tissue type), which allows the surgical hub 106, 206 to infer that the segmental resection portion of the procedure is being performed.

[0149] A node dissection step is then performed in a twelfth step 2524. The surgical hub 106, 206 may infer that the surgical team is dissecting the node and performing a leak test based on data received from the generator indicating that an RF or ultrasonic instrument is being fired. For this particular procedure, the RF or ultrasonic instrument used after the parenchymal tissue is transected corresponds to the node dissection step, thereby enabling the surgical hub 106, 206 to make the above inference. It is noted that the surgeon will periodically alternate between the surgical stapling / cutting instrument and the surgical energy (i.e., RF or ultrasonic) instrument depending on the particular step in the procedure, as different instruments are better suited for certain tasks. Thus, the particular sequence in which the stapling / cutting instrument and the surgical energy instrument are used may indicate which step of the procedure the surgeon is performing. Additionally, in certain instances, a robotic tool may be used for one or more steps in the surgical procedure, and / or a handheld surgical instrument may be used for one or more steps in the surgical procedure. The surgeon(s) can, for example, alternate between using robotic tools and handheld surgical instruments and / or use the devices simultaneously. Once the twelfth step 2524 is completed, the incisions are closed and the post-operative portion of the procedure begins.

[0150] In a thirteenth step 2526, the patient is worked on emerging from anesthesia. The surgical hub 106, 206 may estimate that the patient is emerging from anesthesia based on, for example, ventilator data (i.e., the patient's breathing rate begins to increase).

[0151] Finally, a fourteenth step 2528 is for medical personnel to remove the various patient monitoring devices from the patient. Thus, the surgical hub 2106, 2236 can presume that the patient is being transferred to a recovery room when the hub loses EKG, BP, and other data from the patient monitoring devices. As can be seen from the description of this exemplary procedure, based on data received from various data sources communicatively coupled to the surgical hub 2106, 2236, the surgical hub 2106, 2236 can determine or estimate when each step of a given surgical procedure is occurring.

[0152] Situational awareness is further described in various U.S. patent applications incorporated herein by reference in this disclosure, which are incorporated herein by reference in their entireties. In certain examples, the operation of a robotic surgical system, including, for example, the various robotic surgical systems disclosed herein, may be controlled by the hub 2106, 2236 based on its situational awareness and / or feedback from its components and / or based on information from the cloud 2104 (FIG. 17).

[0153] Imaging system using fusion images One problem inherent in surgical procedures in which the surgeon relies on an imaging system 142 (FIG. 2) is obstructions to the camera 144 (FIG. 2) that adversely affect the ability of the imaging system 142 to visualize the surgical site and thus the surgeon's ability to perform the surgical tasks required for the procedure. Obstructions may include, for example, fluids (e.g., blood) on the lens of the camera 144, on the surface of the body cavity or otherwise present at the surgical site, smoke generated by the electrosurgical instrument or other aerosols present within the body cavity, and / or tissue or other structures overlying the target tissue or structure. Surgical systems may be configured to utilize a variety of imaging techniques to compensate for obstructions, including multispectral imaging of subregions of the FOV of the camera 144, interpolation of subregions of previous image frames captured by the camera 144, comparative multispectral analysis of captured images, and the like.

[0154] In one general aspect, the present disclosure relates to a surgical system configured to utilize segments of images captured at a sampling rate via an imaging system 142a multispectral light source to minimize visualization obstructions due to various obstructions (e.g., surgical smoke). In one aspect, the surgical system can be configured to combine hyperspectral imaging with visible light imaging to minimize image interference caused by obstructions. The surgical system can be configured to detect aspects of the surgical instrument, surgical site, or surrounding substrate or obstructed portions, for example, by utilizing distinct wavelengths or wavelength ranges of EMR. For example, the surgical system can utilize frames from a continuous scanning device to transmit distinct wavelengths of EMR and can include a hyperspectral imaging device configured to scan both inside and outside the visible light spectrum, or a second imaging system configured to emit EMR at a different length than the first or primary imaging system. Thus, the surgical system can be configured to identify occluded portions of image frames at a particular wavelength or set of wavelengths, and to interpolate or replace portions of the occluded image portions with unobstructed image portions of image frames acquired at other EMR wavelengths in order to provide a user with a fully visualized, unobstructed image of the surgical site.

[0155] In aspects utilizing an imaging system including a hyperspectral imager, the hyperspectral imager can scan at a particular rate (e.g., 240 frames per second) that allows a portion of the emitted scan to include EMR from a near infrared or UV laser source. Because EMR at these wavelengths is not affected by obstructions such as surgical smoke, fluids, etc. in the same way as visible light, the hyperspectral imager can be utilized to obtain images of shapes, contours, or features present in both the hyperspectral image and the corresponding visible light image. A control system for the surgical system, such as the control system 133 shown in FIG. 2, can then be configured to replace the obstructed portion of the image obtained utilizing visible light with the corresponding detected hyperspectral feature or image portion to complete the visualization for the surgeon. As another example, the imaging system can include an adjustable EMR source (e.g., spectral light source 150) that is controllable by the control system 133 to emit EMR at a wavelength or set of wavelengths (e.g., within the visible blue-green wavelength range) where absorption of EMR by water is minimal, as obstruction by water or water-containing fluids is particularly likely during a surgical procedure. As another example, the surgical system can further include a second imaging system in addition to the first primary imaging system (e.g., imaging system 142 shown in FIG. 2). In this example, the first imaging system 142 can be configured for imaging in the visible or near-visible EMR spectrum, and the second imaging system can be configured for imaging in a different wavelength spectrum (e.g., long-wave IR (LWIR)). Thus, the second imaging system can be activated or otherwise utilized by the surgical system as needed when the first imaging system is obstructed. In these various aspects, the surgical system minimizes the amount of cleaning required for the camera 144 (e.g., to clear obstructions from the image sensor 135 or other scanning array) and prevents temporary loss of visibility of the surgical field due to obstructions (e.g., surgical smoke or insufflation gases) between the camera 144 and the surgical field.

[0156] In certain examples, the imaging or visualization system is described as including a hyperspectral imaging device or utilizing hyperspectral imaging techniques. However, it should be noted that hyperspectral imaging is one particular type of multispectral imaging. In hyperspectral imaging, the wavelength "bins" are contiguous, so that the hyperspectral imaging techniques utilize the entire EMR spectrum. Conversely, multispectral may mean that the "bins" are separated. In other words, the multispectral imaging system may sense EMR in, for example, the visible, mid-wave IR (MWIR), and LWIR portions of the EMR spectrum (e.g., there may be gaps in the near-infrared (NIR) portion of the EMR spectrum and / or between the MWIR and LWIR portions that the multispectral imaging system does not sense). The imaging or visualization systems and methods described herein should not be construed as limited to any particular examples, including examples describing hyperspectral imaging. Indeed, the imaging or visualization systems and methods may broadly utilize any multispectral imaging devices and techniques.

[0157] To aid in understanding the aforementioned systems and methods, various examples are described within the context of a video-assisted thoracoscopic surgery (VATS) procedure. However, it should be understood that this is for illustrative purposes only and that the described systems and methods are applicable to other situations and / or surgical procedures. A VATS procedure is a surgical procedure in which one or more surgical instruments and one or more thoracoscopes (i.e., cameras) are inserted into a patient's thoracic cavity through a slit positioned between the patient's ribs. The cameras are utilized to provide the surgeon with a view of the inside of the patient's thoracic cavity, allowing the surgeon to properly position / move the surgical instruments and manipulate tissues / structures within the thoracic cavity. Thus, FIG. 22 is an illustration of a surgical system 3000 during the performance of a surgical procedure on a lung 3010, according to at least one embodiment of the present disclosure. The surgical system 3000 for performing a video-assisted surgical procedure can include a variety of different surgical devices, including an imaging device 3002, a grasper 3004, an electrosurgical instrument 3006, or another surgical instrument, and a smoke evacuation device 3008. Additionally, the surgical system 3000 may include or be coupled to a surgical hub 2106, 2236 (FIGS. 17-19), a visualization system 2108 (FIGS. 17-19), or an imaging system 142 (FIG. 2), a control system 133 (FIG. 2), a robotic system 2110 (FIGS. 17-19), and any other system or device described herein. The imaging device 3002 may include a camera 144 (FIG. 2), a spectral light source 150 (FIG. 2), a structured light source 152 (FIG. 2), any other imaging emitter or receiver described herein, or combinations thereof. The imaging device 3002 may be configured to capture and provide images or video of the surgical site within the FOV 3020 to a display screen (e.g., display 146 as in FIG. 2) for viewing by a user (e.g., a surgeon). The imaging device 3002 can be configured to sense EMR within or outside the visible light portion of the EMR spectrum and thereby visualize tissue and / or structures that are either visible or invisible to the naked eye. Based on the visualization provided by the imaging system 142 associated with the imaging device 3002, the surgeon can then control surgical equipment to manipulate the tissue and / or structures to perform the surgical procedure.

[0158] During a surgical procedure, various obstructions, such as surgical plume 3014 or other aerosols, fluids, gases, tissues, structures, etc., may move across the FOV 3020 of the imaging device 3002, thereby preventing the imaging system 132 from being able to fully visualize the surgical site, which in turn may adversely affect the surgeon's ability to perform the procedure. Many surgical systems 3000 include a smoke evacuator 3008 to remove surgical plume 3014, other aerosols, and gases from the body cavity during surgery. However, the smoke evacuator 3008 may not be sufficient to remove all obstructions, or there may be a delay associated with removing the obstructions that prevents the surgeon from properly visualizing the surgical site. Thus, there is a need for systems and methods to compensate for the presence of obstructions and enable visualization of the surgical site through those obstructions.

[0159] In one aspect, an imaging system, such as the imaging system 142 shown in FIG. 2, may be configured to utilize hyperspectral imaging and image fusion techniques to enable visualization through obstructions. For example, FIG. 23 is an illustration of an imaging device 3002 facing multiple obstructions. In this example, the target of the surgical procedure is a subsurface tumor 3038. However, to actually visualize the tumor 3038, the imaging device 3002 must compensate for several different obstructions, including fluid 3030 present on the lens of the imaging device 3002, surgical smoke 3032 present within the body cavity, blood 3034 on the surface of the tissue 3036, the tissue 3036 itself, and structures 3040 located throughout the tissue 3036. In one aspect, the imaging device 3002 may be a hyperspectral imaging device configured to sense EMR across a wavelength spectrum. EMR interacts differently with various objects at different wavelengths. In particular, certain wavelengths of EMR may not be absorbed by certain obstructions at certain wavelengths or wavelength ranges. Thus, by sensing EMR in multiple portions of the EMR spectrum, the imaging system 142 can visualize through an obstruction by sensing EMR at wavelengths that are not absorbed by the obstruction. Additionally, the wavelengths sensed by the imaging device 3002 can be selected to sense wavelengths that are non-interactive (or substantially non-interactive) with typical or expected obstructions. In the illustrated example, the imaging device 3002 can be configured to sense EMR in the visible, MWIR, and LWIR portions of the EMR spectrum.

[0160] In one aspect, the control system can be configured to utilize multispectral (e.g., hyperspectral) imaging to visualize the surgical site in multiple portions of the EMR spectrum, and then provide the user with an unobstructed visualization by replacing an obstructed portion of an image captured in one wavelength range with a corresponding portion of an image captured in another portion of the wavelength range that is not absorbed by the obstruction. An example of such an algorithm is shown in FIG. 24, which is a logic flow diagram of a process 3050 for generating a fusion image utilizing a multispectral EMR source. In the following description of the process 3050, please also refer to FIGS. 2 and 23. The process 3050 can be embodied as, for example, instructions stored in a memory 134 coupled to the control circuit 132 that, when executed by the control circuit 132, causes the control circuit 132 to perform the recited steps of the process 3050. For simplicity, the process 3050 is described as being executed by the control circuit 132, but it should be understood that the process 3050 can be performed by other combinations of hardware, software, and / or firmware.

[0161] Thus, control circuitry 132 executing process 3050 can cause imaging system 142 to sense 3052 EMR (e.g., via imager 3002) in a first wavelength range (e.g., visible light) from the surgical site and then generate 3054 a corresponding first image therefrom. Correspondingly, control circuitry 132 can cause imaging system 142 to sense 3056 EMR (e.g., via imager 3002) in a second wavelength range (e.g., MWIR or LWIR) from the surgical site and then generate 3058 a corresponding second image therefrom.

[0162] Thus, the control circuitry 132 can determine 3060 whether the first image is at least partially occluded. The control circuitry 132 can be configured to make this determination by utilizing object recognition and other computer vision techniques to detect obstructions. If the first image is not at least partially occluded, the process 3050 proceeds along the NO branch and the control circuitry 132 can sense 3052, 3054 EMR, and generate 3054, 3058 a corresponding image, as described above. If the first image is at least partially occluded (i.e., an obstruction is present in the image), the process 3050 proceeds along the YES branch and the control circuitry 132 can generate 3062 a third image by replacing the occluded portion of the first image with the corresponding portion of the second image. If the second wavelength range is selected to not be absorbed by the obstruction, the corresponding portion of the second image will be unobstructed. Thus, the third image provides an unobstructed visualization of the surgical site for the surgeon to view.

[0163] For simplicity, the process 3050 is described in the context of generating and combining two images captured in two different wavelength ranges, although the imaging system 142 can be configured to sense and generate images in any number of wavelength ranges. Figure 23, for example, illustrates an implementation that generates a resultant image by combining image data from at least three different EMR wavelength ranges. Each of the illustrated first image 3042a, second image 3042b, third image 3042c, and fourth image 3042d includes an array of pixels 3043 that collectively visualize the surgical site in a corresponding EMR wavelength range. In this example, a first image 3042a is captured utilizing the visible light portion of the EMR spectrum and includes a first unobstructed portion 3044a, with the remaining portion of the image 3042a being obstructed, a second image 3042b is captured utilizing the MWIR portion of the EMR spectrum and includes a second unobstructed portion 3044b, and a third image 3042c is captured utilizing the LWIR portion of the EMR spectrum and includes a third unobstructed portion 3044c. The control system 133 may also be configured to perform various image processing techniques on the various images generated to improve the visualization provided thereby. For example, a fourth image 3042d is also captured utilizing the visible light portion of the EMR spectrum and thus may correspond to the first image 3042a, but includes additional image processing to identify the fluid (water) obstructed portion 3044d. Thus, the corresponding portion of the first image 3042a may be filtered at a corresponding wavelength or wavelength range (e.g., the blue-green portion of the visible light spectrum) to remove obstructions. Thus, the control circuitry 132 executing the process 3050 may be configured to generate a combination or fusion image 3070 from the initial images 3042a, 3042b, 3042c, 3042d described above.The fused image 3070 may include a first portion 3072 corresponding to an unobstructed portion 3044a of a first image 3042a generated from the visible light portion of the EMR spectrum, a second portion 3074 corresponding to an unobstructed portion 3044b of a second image 3042b generated from the MWIR portion of the EMR spectrum, a third portion 3076 corresponding to an unobstructed portion 3044c of a third image 3042c generated from the LWIR portion of the EMR spectrum, and a fourth portion 3078 corresponding to an obscured portion 3044d of an image generated from the visible light portion of the EMR spectrum, but post-processed to remove the blue-green portion of the visible light spectrum. Each of the aforementioned image portions 3072, 3074, 3076, 3078 may be fused together by the control system 133 to generate the fused image 3070 providing an unobstructed visualization of the tumor 3038 and any other associated structures 3040.

[0164] Another technique that can be utilized to compensate for occlusions present at the surgical site is imaging sub-region interpolation, whereby portions of an image that are occluded, damaged, or otherwise interfered with can be replaced by corresponding portions of an image from a synchronized image set. For example, the surgical control system can utilize a lucky region fusion (LRF) technique to enhance the quality of visualization provided to the user by using multiple image frames. In one aspect, the control system can be configured to provide an occlusion-free visualization to the user by replacing occluded portions of an image with unoccluded portions of a previously captured image. An example of such an algorithm is shown in FIG. 26, which is a logic flow diagram of a process 3100 for generating a fused image utilizing multiple image frames. In the following description of the process 3100, please also refer to FIGS. 2 and 27-29. The process 3100 can be embodied as, for example, instructions stored in a memory 134 coupled to the control circuit 132 that, when executed by the control circuit 132, causes the control circuit 132 to perform the recited steps of the process 3100. For simplicity, the process 3100 is described as being performed by the control circuit 132, but it should be understood that the process 3100 can be performed by other combinations of hardware, software, and / or firmware.

[0165] Thus, the control circuitry 132 executing the process 3100 may generate 3102 an image of the surgical site (e.g., via the imaging system 142) as described above, and then determine 3104 whether the image is at least partially occluded. For example, FIG. 27 is an illustration of a series 3150 of n image frames 3160 captured by the imaging system 142. The nth image frame 3160 may be the last captured image frame 3160, the (n-1)th image frame 3160 may be the image frame 3160 captured immediately before, and so on. Each of the image frames 3160 includes a number of pixels 3151, which may or may not correspond to pixels or cells of the image sensor 135, for example. As seen in FIG. 27, the image frame 3160 may include an unobstructed portion 3162 and an occluded portion 3164. Specifically, when evaluating the nth image frame 3160, the control circuitry 123 executing the process 3100 determines that the nth image frame 3160 is at least partially occluded because it includes an occluded portion 3164 of pixel 3151.

[0166] If the control circuitry 132 determines 3104 that the image is at least partially unobstructed, the process 3100 proceeds along the NO branch and the control circuitry 132 can cause the imaging system 142 to continue generating images for visualization of the surgical site (i.e., visualizing the surgical site) as described above. If the control circuitry 132 determines 3104 that the image is at least partially constructed (e.g., as shown in the nth image frame 3160), the process proceeds along the YES branch and the control circuitry 132 can retrieve 3106 a previous image from the image set 3150. In one aspect, the control circuitry 132 can successively retrieve 3106 one or more previous images from the image set 3150 until the control circuitry 132 locates a corresponding unobstructed image portion that replaces the occluded portion of the first image.

[0167] Thus, the control circuitry 132 may generate 3108 an updated image from the original image and one or more previous images taken from the image set 3150. For example, FIGS. 28 and 29 show an updated or fused image 3152 generated from a number of consecutive image frames 3160. In this particular example, n is equal to 60, but this is for illustrative purposes only. In FIG. 28, the number shown within each pixel 3151 corresponds to the image frame 3160 from which the particular pixel 3151 was extracted. As can be seen, the fused image 3152 is generated from a combination of pixels 3151 across several different image frames 3160. Specifically, as shown in FIG. 27, image frames 3160 55-60 correspond to the (n-5)th through nth image frames 3160, respectively, in sequence. Thus, the control circuitry 132 can be configured to repeatedly retrieve 3106 previous images from the image set 3150 captured by the imaging system 142 and extract image portions, such as pixels 3151 that are unoccluded in the retrieved image but that correspond to pixels 3151 that are occluded in the successive images. The control circuitry 132 can repeat this process until a completely or substantially unoccluded collection of image portions from the image set 3150 is retrieved and then fuses the image portions together to generate 3108 an updated image. The resulting fused image 3152 generated using this technique is shown in FIG. 29, illustrating how structures 3140, such as tumors 3038 and blood vessels, are made visible to the user from the initial partially occluded image.

[0168] Another technique that can be utilized to compensate for occlusions present at the surgical site is to perform a comparative analysis of a set of synchronized imaging devices. The control system 133 can be configured to interlace multiple image portions generated by multiple synchronized imaging devices to generate a fused image. In particular, a portion of an image generated by a first or primary imaging system (e.g., imaging system 142 shown in FIG. 2) can be replaced with a corresponding portion of an image generated by a secondary imaging system. In particular, the set of imaging systems can be configured to time-index their scans. An occluded, corrupted, unclear, or otherwise interfered with portion of a first scan generated by a first imaging system can be replaced with a clearer and / or verified portion of a second scan (time-indexed according to the first scan) generated by a second imaging system. If image data is missing, corrupted, or obstructed in the imaging of the primary dynamic dataset generated by the first imaging system, a secondary scan from another imaging system (capable of sensing at another wavelength or wavelength range) may be utilized by the control system 133 to sharpen, replace, or interpolate the primary image to improve visualization of the surgical site for the user.

[0169] Surgical system control based on multiple sensed parameters - Patents.com One challenge inherent to any surgical procedure and surgical instrument is to control the surgical instrument in a manner ideal for a given patient and / or tissue condition. To that end, some surgical instruments include sensors for sensing various parameters associated with the surgical instrument and / or the tissue being manipulated by the surgical instrument. However, some sensed data may indicate different conditions or status of the tissue and therefore may be inconclusive in the absence of additional data. Thus, the surgical system may incorporate data from an imaging system along with other sensed data to resolve ambiguities and ideally control the surgical instrument according to the determined state / status of the tissue.

[0170] In one general aspect, the present disclosure relates to a control system configured to utilize two related but non-identical sources of data to distinguish between different conditions of tissue being acted upon by a surgical instrument. Such conditions include, for example, fluid flow within the tissue and the thermal effects of energy directed by the surgical instrument on the tissue. The control system can be configured to control a surgical instrument, such as the surgical instrument 3290 described below.

[0171] 30 is a schematic diagram of a surgical instrument 3290 configured to control various functions, according to at least one embodiment of the present disclosure. In one embodiment, the surgical instrument 3290 is programmed to control distal translation of a displacement member, such as a closure member 3264. The surgical instrument 3290 includes an end effector 3292 that may include a clamp arm 3266, a closure member 3264, and an ultrasonic blade 3268 that may replace or operate in conjunction with one or more RF electrodes 3296 (shown in dashed lines). The ultrasonic blade 3268 is coupled to an ultrasonic transducer 3269 that is driven by an ultrasonic generator 3271.

[0172] In one aspect, the sensor 3288 may be implemented as a limit switch, an electromechanical device, a solid-state switch, a Hall effect device, an MR device, a GMR device, a magnetometer, among others. In other implementations, the sensor 3288 may be a solid-state switch that operates under the influence of light, such as a light sensor, an IR sensor, an ultraviolet sensor, among others. Additionally, the switch may be a solid-state device, such as a transistor (e.g., FET, junction FET, MOSFET, bipolar, etc.). In other implementations, the sensor 3288 may include a non-electrical conductor containing switch, an ultrasonic switch, an accelerometer, and an inertial sensor, among others.

[0173] In one aspect, the position sensor 3284 may be implemented as an absolute positioning system including a magnetic rotary absolute positioning system implemented as an AS5055EQFT single chip magnetic rotary position sensor available from Austria Microsystems, AG. The position sensor 3284 may be interfaced with the control circuit 3260 to provide an absolute positioning system. The position may include multiple Hall effect elements located above a magnet and coupled to a CORDIC processor, also known as the digit-by-digit method and the Boulder algorithm, which is provided to implement simple and efficient algorithms for calculating hyperbolic and trigonometric functions, requiring only addition, subtraction, bit shifting and table lookup operations.

[0174] In some examples, the position sensor 3284 may be omitted. If the motor 3254 is a stepper motor, the control circuit 3260 can track the position of the closure member 3264 by summing the number and direction of steps the motor is commanded to take. The position sensor 3284 may be located in the end effector 3292 or in any other portion of the instrument.

[0175] The control circuit 3260 can be in communication with one or more sensors 3288. The sensor 3288 may be positioned on the end effector 3292 and adapted to operate with the surgical instrument 3290 to measure various derived parameters such as gap distance versus time, tissue compression versus time, and anvil strain versus time. The sensor 3288 may include a magnetic sensor, a magnetic field sensor, a strain gauge, a pressure sensor, a force sensor, an inductive sensor such as an eddy current sensor, a resistive sensor, a capacitive sensor, an optical sensor, and / or any other suitable sensor for measuring one or more parameters of the end effector 3292. The sensor 3288 may include one or more sensors.

[0176] An RF energy source 3294 is coupled to the end effector 3292 and applied to the RF electrode 3296 when the RF electrode 3296 is provided in the end effector 3292 in place of the ultrasonic blade 3268 or when the RF electrode 3296 is provided to operate in conjunction with the ultrasonic blade 3268. For example, the ultrasonic blade may be made of a conductive metal and used as a return path for the electrosurgical RF current. The control circuit 3260 controls the delivery of RF energy to the RF electrode 3296.

[0177] Additional details are disclosed in U.S. patent application Ser. No. 15 / 636,096, filed Jun. 28, 2017, entitled "SURGICAL SYSTEM COUPLABLE WITH STAPLE CARTRIDGE AND RADIO FREQUENCY CARTRIDGE, AND METHOD OF USING SAME," which is incorporated by reference in its entirety.

[0178] In various aspects, the sensor 3288 of the surgical instrument 3290 can include a sensor configured to detect or measure various electrical parameters associated with the tissue acted upon by the surgical instrument 3290, such as tissue capacitance or impedance. In various aspects, the sensor 3288 can also include a sensor configured to detect or measure various physical parameters associated with the tissue acted upon by the surgical instrument 3290, such as temperature, viscoelastic compression (e.g., tissue creep, time to stability, or rate of initial loading), or thickness (e.g., may be detected upon initial contact of the jaws with the tissue). Additionally, the image sensor 135 of the control system 133 shown in FIG. 2 can be utilized to detect or measure various tissue parameters based on the EMR emitted by the imaging system 142 using the various techniques described above. For example, the image sensor 135 can be configured to detect the refractive index of the tissue at various wavelengths, the polarization of the EMR / light reflected by the tissue, passive IR emission by the tissue, or a Doppler wavelength shift associated with the tissue. Any of these imaging-based parameters may be utilized in conjunction with other sensed parameters (e.g., electrical or physical parameters) to ascertain a tissue condition or status that cannot be directly ascertained via the parameters individually.

[0179] In one aspect, the control system may be configured to control one or more operating parameters associated with the surgical system based on the state or condition of the tissue acted upon by the surgical instrument, which may be determined based on parameters sensed by the imaging system and other sensors. An example of such an algorithm is shown in FIG. 31, which is a logic flow diagram of a process 3300 for controlling a surgical system according to sensed parameters. In the following description of the process 3300, please also refer to FIGS. 2 and 30. The process 3300 may be embodied as, for example, instructions stored in a memory 134 coupled to the control circuit 132 that, when executed by the control circuit 132, cause the control circuit 132 to perform the recited steps of the process 3300. For simplicity, the process 3300 is described as being executed by the control circuit 132, but it should be understood that the process 3300 may be performed by other combinations of hardware, software, and / or firmware.

[0180] Thus, the control circuitry 132 executing the process 3300 may receive 3302 a measurement of a first tissue parameter via the imaging system 142. As mentioned above, the first tissue parameter may include, for example, the refractive index of the tissue at various wavelengths, the polarization of light reflected by the tissue, passive IR emission by the tissue, or a Doppler wavelength shift associated with the tissue.

[0181] Thus, the control circuitry 132 may receive 3304 a measurement of the second tissue parameter via the sensor 788. As previously mentioned, the second tissue parameter may include various electrical and / or physical properties of the tissue, such as, for example, temperature, viscoelastic compression, or thickness of the tissue.

[0182] Thus, the control circuitry 132 can determine (3306) the state or condition of the tissue based on a combination of measurements of the received 3302, 3304 tissue parameters and then control (3308) the surgical instrument 3290 accordingly. The same measurements of various electrical and / or physical properties of the tissue may indicate different conditions of the tissue, which in turn may require different control adjustments to be applied to the surgical instrument 3290. Without additional or supplemental information, the control system 133 may not be able to correctly control the surgical instrument 3290 for a given state of the tissue based on measurements of various electrical and / or physical properties of the tissue in situations where the tissue state is ambiguous. Thus, the control system described herein supplements the electrical and / or physical properties sensed by the sensor 3288 with tissue parameters sensed via the imaging system 142 to accurately ascertain the state or condition of the tissue and then appropriately control the surgical instrument 3290. For example, various methods of controlling the surgical instrument 3290 in response to detecting an increase in tissue temperature localized to the end effector 3292 of the surgical instrument 3290 (i.e., the second tissue parameter 3304 received during process 3300) may be appropriate. If the control system 133 detects a corresponding change in the polarization or refractive index of the tissue (i.e., the first tissue parameter 3302 received during process 3300), the control circuitry 132 may determine 3306 that the tissue is experiencing collateral thermal damage and control 3308 the surgical instrument 3290 to reduce the instrument power level or provide a suggestion to the user to reduce the instrument power level. Conversely, if no corresponding change in the polarization or refractive index of the tissue is detected, the control circuitry 132 may determine 3306 that the tissue is not experiencing collateral thermal damage and control 3308 the surgical instrument 3290 to maintain or increase the instrument power level or provide a suggestion to this effect. As another example, various methods of controlling the surgical instrument 3290 may be appropriate in response to detection of tissue impedance of tissue grasped by the end effector 3292 of the surgical instrument 3290 (i.e., the second tissue parameter 3304 received during the process 3300).If the control system 133 does not detect a change in tissue impedance while the imaging system 142 visualizes tissue movement, creep, or compression (i.e., the first tissue parameter 3302 received during process 3300), the control circuit 132 can determine (3306) that there is a subsurface irregularity in the grasped tissue.

[0183] The control system 133 described herein may be implemented or executed, for example, in the surgical instrument 3290, a surgical hub 2236 (FIG. 21) to which the surgical instrument 3290 is communicatively coupled (e.g., an energy device 3241 as shown in FIG. 21), or a combination thereof (e.g., using a distributed processing protocol). When the control system is embodied as a component of the surgical instrument 3290, imaging data may be received either directly from the imaging system 142 or via a surgical hub 2106, 2236 (FIGS. 17-19) that is in turn coupled to the imaging system 142. When the control system 133 is embodied as a component of the surgical hub 2106, 2236, imaging data can be received from the imaging system 142 coupled to the surgical hub 2106, 2236 and surgical instrument sensor data can be received from the surgical instrument 3290 coupled to the surgical hub 2106, 2236 which the control system 133 of the surgical hub 2106, 2236 can then determine and transmit the appropriate surgical instrument control adjustments to the surgical instrument 3290 for execution thereby.

[0184] Adaptive optics system for compensating for imaging artifacts In one aspect, a control system, such as control system 133 described in connection with FIG. 2, may be configured to compensate for imaging artifacts associated with the imaging system 142 coupled thereto. In one aspect, control system 133 may be configured to adjust the optical signals received by imaging system 142 across multiple optical wavelengths in combination with selective imaging segment selection within a sample rate of over 60 Hz to remove optical particulate obstructions from visualization. In one aspect, control system 133 may be configured to emit a projected control beam (e.g., via imaging system 142) and correspondingly monitor the return signal on a separated frame of a scanning array (e.g., of image sensor 135) to determine EMR / optical distortion due to particulates in the gas occupying the body cavity. The variance of the control projection from its projection source gives control system 133 a baseline for adjusting the scope visualization frame in later portions of the scan.

[0185] Surgical system control based on airborne particle characteristics One challenge inherent to surgical procedures using electrosurgical instruments is the smoke generated by the instruments. Surgical smoke may contain toxic gases and vapors, dead and live cellular material, blood fractions, and bioaerosols including viruses, as well as mutagenic and carcinogenic compounds. It is therefore highly desirable to remove these particulates from the surgical site, and thus smoke evacuators are generally utilized in surgical procedures that result in the generation of surgical smoke. However, because different particulate types may require different types of control adjustments to precisely control and mitigate smoke generation during a surgical procedure, it is desirable to control the smoke evacuator and other surgical equipment (including the surgical instrument) according to the type of particulate being generated. The surgical system may, for example, modify the surgical instrument energy profile so that less smoke is generated and / or automatically control the smoke evacuator according to the type of particulate being generated.

[0186] In one general aspect, the present disclosure relates to a control system configured to detect the level of polarization of light emitted by an imaging system to determine parameters of the particulate cloud and adjust control parameters of a linked system or device accordingly. In one further aspect, the polarization of EMR reflected from the detected particulates can be utilized in combination with the vectorization and amount of particulates generated to determine the source of the particulates, which can then be utilized to control the device causing the particulate generation to improve visualization at the surgical site. In a further aspect, the polarization of EMR reflected from the detected particulates can be utilized to determine whether adjusting control parameters of an electrosurgical instrument or smoke evacuator would be more effective in improving visualization at the surgical site.

[0187] 32 is a diagram of a polarized EMR source 3500 for detecting different particulate types, according to at least one embodiment of the present disclosure. The polarized EMR source 3500 can include an emitter 3502 configured to emit EMR 3506 and a polarizing filter 3504 configured to polarize the emitted EMR 3506. The polarizing filter 3504 can be removably securable or integrally attached to the emitter 3502. The polarized EMR source 3500 can be embodied as a component of an imaging system, which can include, for example, the surgical visualization system 100 shown in FIG. 1, the imaging system 142 shown in FIG. 2, and / or the surgical visualization system 500 shown in FIG. 5. Accordingly, the emitter 3502 can include the emitter 106 shown in FIG. 1, the structured light source 152 shown in FIG. 2, the spectral light source 150 shown in FIG. 2, or the like. The imaging system, in turn, may be embodied as a component of a surgical system, such as the robotic surgical system 110 shown in Figure 1, which may further include a control system configured to control various aspects of the surgical system. The control system may include, for example, the control system 133 shown in Figure 2 and / or the control system 600 shown in Figure 11.

[0188] During a surgical procedure, airborne particulates may be present at the surgical site. These particulates may include both naturally occurring particulates and non-natural or synthetic particulates. Naturally occurring particulates may be generated by interaction between a surgical instrument, such as an electrosurgical instrument, and the tissue being treated. Naturally occurring particulates may include, for example, dead and live cellular material, blood fractions, and other biomaterials. Man-made or synthetic particulates may be introduced to the surgical site by surgical staff. These particulates may be embodied as smoke or aerosols present in or at the surgical site. Generally speaking, the presence of such particulates may be undesirable, and therefore many surgical systems include smoke evacuators to remove undesirable smoke or aerosols from the surgical site. However, the imaging system may be configured to detect particulates (i.e., smoke) generated at the surgical site, and the control system may be configured to control various operating parameters of the surgical system or its components based on the characteristics or attributes of the detected particulates. Several examples of such control algorithms are described herein.

[0189] Referring back to FIG. 32, as discussed above, airborne particulates at a surgical site may include both naturally occurring particulates 3510 and artificial particulates 3512. It may be beneficial to be able to distinguish between different types of particulates present at a surgical site, since different actions may be required to mitigate the presence of each of the different types of particulates. For example, if the particulate type detected is naturally occurring particulates 3510, which may be generated from an electrosurgical instrument treating tissue, it may be desirable to control the electrosurgical instrument to mitigate the generation of naturally occurring particulates 3510 (e.g., by reducing the energy duty cycle or otherwise altering the energy delivery profile of the instrument). Conversely, if the particulate type detected is synthetic particulates 3512, control of the electrosurgical instrument will not affect the presence of synthetic particulates 3512, since that particulate type is not generated by operation of the electrosurgical instrument. Instead, it may be desirable to increase the suction flow rate of the smoke evacuator to remove the synthetic particulates 3512 from the surgical site. Additionally, if a combination of different particulate types is detected at a surgical site, it may be desirable to control the electrosurgical instrument and the smoke evacuator in combination with each other, with different control adjustments for each device. Thus, the control system of the surgical system can be configured to control various devices or components of the surgical system to detect different types of airborne particulates present at the surgical site and to mitigate or eliminate the particulates from the surgical site.

[0190] In one aspect, the naturally occurring particulates 3510 and the synthetic particulates 3512 can be distinguished from one another based on the reflective properties of the airborne particulates 3510, 3512 when exposed to polarized EMR 3506. For example, the control system 133 can be configured to cause the emitter 3502 to pulse coherent EMR at multiple different wavelengths with or without polarization to determine the distance to the body undergoing the surgical procedure and define a range gate such that the control system 133 obtains depolarized measurements only from EMR 3506 reflected from particulates in the air space between the emitter 3502 and the body cavity and not from the body cavity itself. In particular, the control system 133 can be configured to cause the emitter 3502 to pulse coherent EMR at a first wavelength and a second wavelength. The first wavelength can be selected such that the EMR at the first wavelength is substantially non-interactive with the naturally occurring particulates 3510 and the synthetic particulates 3512 and can therefore be reflected from the body cavity through the smoke. The control system 133 can then determine the distance to the body cavity via the time-of-flight sensor system 1104, 1204 according to the time difference between when the EMR is emitted and when the reflected EMR is detected, for example, as described in connection with Figures 14-16. The second wavelength can be selected such that the EMR at the second wavelength is substantially interactive with the naturally occurring particulates 3510 and the synthetic particulates 3512 and can thus be utilized to detect or measure characteristics associated with different particulate types. Thus, the control system 133 can utilize the body cavity distance determined by pulsing the EMR at the first wavelength to range gate the measurements received by the EMR at the second wavelength to ensure that only measurements of airborne particulates are being taken. The control system 133 can then determine whether the airborne particulate is a naturally occurring particulate 3510 and / or a synthetic particulate 3512 based on the reflective characteristics of the airborne particulates 3510, 3512 and control other components of the surgical system accordingly.

[0191] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system based on the type of airborne particulate detected at the surgical site. An example of such an algorithm is shown in FIG. 33A, which is a logic flow diagram of a process 3600 for controlling the surgical system according to a detected particulate type, and FIG. 33B is a logic flow diagram of a process 3650 for controlling the surgical system according to a detected particulate type detected within a defined range gate. In the following description of the processes 3600, 3650, please also refer to FIG. 2. The processes 3600, 3650 can be embodied as, for example, instructions stored in the memory 134 coupled to the control circuit 132 that, when executed by the control circuit 132, cause the control circuit 132 to perform the recited steps of the processes 3600, 3650. For simplicity, the processes 3600, 3650 are described as being executed by the control circuit 132, but it should be understood that the processes 3600, 3650 can be performed by other combinations of hardware, software, and / or firmware.

[0192] 33A, the control circuitry 132 executing a process 3600 can cause the imaging system 142 to emit 3602 polarized EMR directed toward the surgical site, for example, via the polarized EMR source 3500.

[0193] Thus, the control circuitry 132 can receive 3604 polarized EMR reflected from airborne particulates at a surgical site (e.g., within a body cavity) and determine 3606 whether the detected particulate type is a naturally occurring particulate 3510 or a man-made particulate 3512. The control circuitry 132 can distinguish between different types of airborne particulates 3510, 3512 due to their different reflectance characteristics when exposed to polarized EMR. In particular, one of the types of airborne particulates 3510, 3512 (e.g., man-made particulates 3512) can scatter polarized EMR at a higher rate than the other (e.g., naturally occurring particulates 3510). This can reduce the degree of visualization of the scattered airborne particulate type or otherwise affect the manner in which the reflected EMR is received by the image sensor 135 of the imaging system 142. Thus, this difference in visualization of the different types of airborne particulates 3510, 3512 can be characterized and utilized to identify the type of airborne particulates 3510, 3512 present at a surgical site (e.g., within a body cavity).

[0194] Thus, if the particulate is a naturally occurring particulate 3510, the process 3600 may proceed along the YES branch and the control circuitry 132 may adjust 3608 the control parameters of the surgical system to a first state corresponding to a naturally occurring particulate 3510. Conversely, if the particulate is an artificial particulate 3512, the process 3600 may proceed along the NO branch and the control circuitry 132 may adjust 3610 the control parameters of the surgical system to a second state corresponding to an artificial particulate 3512.

[0195] In another embodiment, there may be various combinations of naturally occurring and artificial particulates 3510 present within the body cavity. In such cases, the control circuitry 132 can instead determine the relative ratios of types of airborne particulates 3510, 3512 present within the body cavity (e.g., due to the relative degree to which visualization is reduced or affected) and then control the surgical device or combination of surgical devices accordingly.

[0196] In yet another embodiment, the control system can be configured to range gate the measurements and / or visualization based on the polarized EMR source 3500. Such an embodiment can be embodied by the process 3650 shown in FIG.

[0197] Thus, the control circuitry 132 executing the process 3650 can cause the imaging system 142 to emit 3652 EMR at a first wavelength directed toward the surgical site, for example, via the structured light source 152 and / or the spectral light source 150. In one aspect, the first wavelength can be a wavelength that is substantially non-interactive with the naturally occurring particulates 3510 and synthetic particulates 3512 imaged by the imaging system 142.

[0198] Thus, the control circuit 132 receives 3654 EMR reflected from the body cavity (i.e., the surgical site) via the imaging system 142 and defines 3656 a range gate corresponding to the space between the emitter of the imaging system 142 and the body cavity surface, as described above.

[0199] Thus, the control circuitry 132 can cause the imaging system 142 to emit 3658 polarized EMR at a second wavelength directed toward the surgical site, for example, via the polarized EMR source 3500. In one aspect, the second wavelength can be a wavelength that is substantially interactive with the naturally occurring particulates 3510 and synthetic particulates 3512 imaged by the imaging system 142.

[0200] Thus, the control circuitry 132 can receive, via the imaging system 142, polarized EMR reflected within a defined range gate that can correspond to airborne particles 3510, 3512 located between an emitter of the imaging system 142 and the body cavity surface. The control circuitry 132 can then determine 3662 whether the detected particle type is a naturally occurring particle 3510 or a man-made particle 3512, and can adjust 3608, 3610 the control parameters of the surgical system to a first state or a second state, as described above with respect to the process 3600 shown in FIG.

[0201] In another aspect, the surgical system can be further configured to track the movement of airborne particles throughout the course of a surgical procedure, which can then be utilized to characterize the movement and changes in size or configuration of the cloud defined by the airborne particles. By characterizing the movement of airborne particles over time, the surgical system can, for example, determine how well the body cavity is evacuated and then adjust or provide recommendations to the user to adjust the location or size of the smoke evacuation or insufflation. For example, the surgical system can stop the first smoke evacuator or first insufflator and activate the second smoke evacuator or second insufflator to adjust the gas circulation current in the body cavity, thereby improving smoke evacuation performance by mitigating any vortices (i.e., areas where the particle and / or gas movement vector is zero or close to zero) in the body cavity. As another example, the surgical system can adjust the motor level or fan level of the smoke evacuator or aspirator to improve smoke evacuation performance.

[0202] In one aspect, the control system 133 can be configured to track and characterize the movement of airborne particulates by characterizing the detection of the particulate across cells or pixels of the image sensor 135. In particular, the control system 133 can determine at which pixels the image sensor 135 detects the particulate and then track the movement of the particulate over time across the pixel array of the image sensor 135. In one aspect, the control system 133 can be configured to divide an image obtained via the image sensor 135 into two or more pixel array sections, generate a motion vector corresponding to a generalized change in position due to the detected airborne particulate from a first time instance to a second time instance, and then characterize the movement or change in composition of the particulate cloud accordingly.

[0203] For example, Figures 34A-C show a pixel array 3700 of the image sensor 135 consisting of a number of pixels 3701. Additionally, Figures 34A-C show the change in detected particle position over time and the generalized particle cloud motion vector calculated therefrom. Note that while pixel array 3700 is depicted as being a 5x5 array, this is for illustrative purposes only, and neither image sensor 135 nor the selected subsection of its pixels is limited to a 5x5 array. In the following description of Figures 34A-C, reference should also be made to Figure 2.

[0204] FIG. 34A illustrates the detection array 3700 at time t1, which shows detection of a first particle 3702a, a second particle 3702b, a third particle 3702c, a fourth particle 3702d, and a fifth particle 3702e at an indicated pixel 3701 of the image sensor 135. FIG. 34B illustrates the detection array 3700 at time t2, which shows that the first and fifth particles 3702a, 3702e do not change position, and the second, third, and fourth particles 3702b, 3702c, 3702d are detected at different pixels 3701 of the image sensor 135. Based on the detected movement of the particles, the control circuit 132 can be configured to determine a vector representation of the movement of each of the pixels. The vector representation can include both direction and magnitude. Based on the direction and magnitude of the motion vector, the control circuitry 132 can be further configured to calculate (e.g., using vector addition) a vector 3704 corresponding to a generalized motion of a cloud defined by the detected particulate. Thus, the control circuitry 132 can track the change of the particulate cloud or aerosol from a first state 3710 (e.g., a first position or a first size) to a second state 3712 (e.g., a second position or a second size), as shown in FIG. 35, according to the vector 3704 calculated from the change in pixels of the image sensor 135 where the particulate is detected.

[0205] In one aspect, control circuitry 132 coupled to image sensor 135 can be configured to track the movement of detected airborne particles, to calculate a generalized motion vector corresponding to changes in position of the detected airborne particles within pixel array 3700 (which can represent the entire pixel array of image sensor 135 or a subsection thereof), and then control various connected surgical devices, such as insufflators, smoke evacuators, and / or surgical instruments accordingly. In one aspect, control system 133 including control circuitry 132 can be embodied as a surgical hub 2106, 2236, as described above under the heading Surgical Hub System. In this aspect, the surgical devices can be communicatively connected (e.g., paired) to surgical hub 2106, 2236 and controlled according to the systems and processes described.

[0206] In another embodiment, the control system 133 can be configured to utilize Raman spectroscopy techniques to determine the vibrational / rotational behavior of airborne particles, for example, using near-infrared, UV, or a combination of near-infrared and UV wavelengths. Data obtained from such techniques can provide information, for example, about the gas-phase species (e.g., benzene vs. aldehyde), which in turn can provide insight into the type of tissue from which the particles were generated or the efficiency of the energy applied to the tissue. Since the desired information of the species is contained in the inelastic scattering of the EMR, the control system 133 can include a filter (e.g., band-pass or notch filter) coupled to the detector, for example, to filter the elastic scattering of the source EMR. The signal generated by the image sensor 135 or another such detector (e.g., a CCD detector) according to the Raman spectroscopy technique can be based on the inherent structural properties of the detected molecules. In particular, Raman spectroscopy is based on the concept that photons emitted by, for example, a suitable emitter excite molecules to a higher energy state, which causes the scattered photons to change frequency as a result of conserving energy from the vibrational / rotational changes of the molecule. This change in frequency of the scattered photons can be used to characterize the type of molecule with which the photons have interacted by comparing the detected signal with pre-characterized data for a given excitation frequency according to the particular type of monochromatic light source used. The determined molecular type of the particulate can be used for several different applications, including providing specific data regarding the relative amounts of potentially dangerous molecules produced at the surgical site for safety monitoring purposes. The determined molecular type of the particulate can also be used to evaluate the effectiveness and health of a smoke evacuation system or its filters.

[0207] Surgical system control based on smoke cloud characteristics. One challenge inherent to surgical procedures using electrosurgical instruments is the smoke generated by the instruments. Surgical smoke may contain toxic gases and vapors, dead and live cellular material, blood fractions, and bioaerosols including viruses, as well as mutagenic and carcinogenic compounds. It is therefore highly desirable to remove these particulates from the surgical site, and thus smoke evacuators are generally utilized in surgical procedures that result in the generation of surgical smoke. However, to precisely control and mitigate smoke generation during surgical procedures, it is desirable to control the smoke evacuator and other surgical equipment (including surgical instruments) according to the amount of smoke at the surgical site, the variation of the smoke cloud over time (e.g., whether the smoke cloud is actively accumulating or decreasing), and other such smoke cloud characteristics. The surgical system can, for example, modify the surgical instrument energy profile to generate less smoke and / or automatically control the smoke evacuator according to the amount of surgical smoke generated.

[0208] In one general aspect, the present disclosure is directed to a surgical system configured to detect and characterize an amorphous three-dimensional particulate cloud generated during a surgical procedure. The surgical system can be configured to detect the movement of the particulate cloud relative to the abdominal cavity and the surgical site, and then control various surgical devices, such as surgical instruments or smoke evacuation devices, accordingly. In one general aspect, the present disclosure is directed to a control system configured to define a surface or boundary of a cloud or particulate cluster generated during a surgical procedure, and to analyze various characteristics of the defined cloud, such as the direction and rate of change of the boundary, to control various control parameters of the surgical system, such as the power level of a surgical instrument / generator or smoke evacuation motor control. In one further aspect, the control system can be configured to generate the boundary by defining a predefined density of particulates based on the overall volume of the particulates or the size of the particulates. In another further aspect, the rate of change of the particulate cloud surface boundary can be utilized to directionally define the rate of change of an energy device or smoke evacuation mechanism.

[0209] 36 is a diagram of a surgical system 3750 during the performance of a surgical procedure in which a particulate cloud 3752 is generated, according to at least one embodiment of the present disclosure. The surgical system 3750 may be embodied as a robotic surgical system, such as, for example, the robotic surgical system 110 shown in FIG. 1. The surgical system 3750 may include an electrosurgical instrument 3754, a smoke evacuator 3756, a grasper 3750, and any other surgical devices for treating, cutting, or otherwise manipulating tissue 3760 for a surgical procedure. Although not shown in FIG. 36, the surgical system 3750 may further include an imaging system, which may include, for example, the surgical visualization system 100 shown in FIG. 1, the imaging system 142 shown in FIG. 2, and / or the surgical visualization system 500 shown in FIG. 5. The surgical system 3750 may further include a control system, which may include, for example, the control system 133 shown in FIG. 2 and / or the control system 600 shown in FIG. 11.

[0210] During a surgical procedure, airborne particulates 3751 may be generated by interactions between a surgical instrument, such as an electrosurgical instrument 3754, and the tissue 3760 being treated. These particulates 3751 may be embodied as smoke or aerosol clouds 3752 present in or at the surgical site. Generally speaking, the presence of such particulates 3751 may be undesirable, and therefore many surgical systems 3750 include a smoke evacuator 3756 to remove the particulates 3751 from the surgical site. However, the imaging system may be configured to image the particulates 3751 and / or smoke generated at the surgical site, and the control system may be configured to control various operating parameters of the surgical system 3750 or its components based on the characteristics or properties of the imaged smoke. Some examples of such control algorithms are described herein.

[0211] In one aspect, the control system can be configured to control one or more operating parameters associated with the surgical system 3750 based on one or more characteristics associated with the smoke cloud generated at the surgical site. An example of such an algorithm is shown in FIG. 37, which is a logic flow diagram of a process 3800 for controlling a surgical system according to particulate cloud characteristics. See also FIG. 2 in the description of process 3800 below. Process 3800 can be embodied as, for example, instructions stored in memory 134 coupled to control circuitry 132 that, when executed by control circuitry 132, cause control circuitry 132 to perform the recited steps of process 3800. For simplicity, process 3800 is described as being executed by control circuitry 132, however, it should be understood that process 3800 can be performed by other combinations of hardware, software, and / or firmware.

[0212] Thus, the control circuitry 132 executing the process 3800 may use any of the techniques described above to detect 3802 the presence of airborne particles within the FOV of the imaging system 142. In general, the image sensor 135 of the imaging system 142 may detect EMR emitted by the structured light source 152 and / or the spectral light source 150 and reflected by the airborne particles to detect / image the particles.

[0213] Thus, the control circuitry 132 can characterize 3804 the particulate cloud defined by the detected particulates. In one aspect, the control circuitry 132 can be configured to define a three-dimensional boundary of the particulate cloud to describe an amorphous three-dimensional construct whose density, volume, location, movement, and / or boundary can be tracked over time. The boundary of the particulate cloud at the surgical site can be defined in a variety of different ways. For example, the particulate cloud boundary can be defined as a volume that encompasses all airborne particulates detected within the FOV of the imaging system 142. As another example, the particulate cloud boundary can be defined as a volume having a threshold density of airborne particulates.

[0214] Thus, the control circuitry 132 can determine 3806 whether one or more characteristics of the particle cloud breach a threshold. Such tracked characteristics can include, for example, the density of the particle cloud, the volume of the particle cloud, the position of the particle cloud and / or its boundaries, the motion of the particle cloud and / or its boundaries, and / or the rate of change or other derivative of any of the aforementioned characteristics. The threshold of the tracked characteristics may depend on other parameters, such as the surgical situation (e.g., the type of surgical procedure being performed). If the threshold is not breached, the process 3800 can proceed along the NO branch and the control circuitry 132 can continue as described above, for example, until a stop criterion is met (e.g., the surgical procedure is completed). If the threshold is breached, the process 3800 can proceed along the YES branch and the control circuitry 132 can continue as described below.

[0215] Thus, the control circuit 132 may adjust 3808 one or more control parameters of the surgical system 3750. Control parameters adjustable by the control circuit 132 may include surgical instrument / generator energy levels, smoke evacuator suction, visualization parameters, etc. For example, FIG. 38 is a series of graphs 3850, 3852, 3854 illustrating adjustment of control parameters based on particulate cloud characteristics by the control circuit 132 executing the process 3800. The first graph 3850 shows a first line 3860 illustrating the change in smoke cloud density, represented by the vertical axis 3856, over time, represented by the horizontal axis 3858. The second graph 3852 shows a second line 3868 illustrating the change in energy duty cycle of the electrosurgical instrument 3754 (or the generator driving the electrosurgical instrument 3754), represented by the vertical axis 3866, over time, represented by the horizontal axis 3858. The third graph 3854 shows a third line 3880 illustrating the change in smoke evacuation or aspiration flow rate of the smoke evacuator 3756, represented by the vertical axis 3878, over time, represented by the horizontal axis 3858. In combination, the graphs 3850, 3852, 3854 show an exemplary predictive implementation of the process 3800 during a surgical procedure, where the process 3800 adjusts 3808 the energy duty cycle of the electrosurgical instrument and smoke evacuator aspiration flow control parameters according to the characterized smoke cloud density.

[0216] Initially, the electrosurgical instrument 3754 does not apply energy to the captured tissue 3760, as indicated by the first diagram 3890. Thus, the energy duty cycle of the electrosurgical instrument 3754 is zero, the smoke evacuator aspiration flow rate is at a base or default rate, and no smoke is being generated (because no energy is being applied to the tissue 3760). At time t1, the surgeon activates the electrosurgical instrument 3754 and begins to apply energy to the tissue 3760, represented by the energy duty cycle increasing 3870 from 0 to E3. The application of energy to the tissue 3760 begins to generate smoke at the surgical site, represented by the smoke cloud density increasing 3862 exponentially from 0 for a period of time after t1. Additionally, in response to the energy being activated, the smoke evacuator flow rate may be increased 3882 by the control circuit 132 from Q1 to Q2 as the smoke evacuator 3756 begins to attempt to remove the generated smoke from the surgical site. At this stage, the control circuitry 132 may begin detecting 3802 the airborne particles produced by the application of energy and characterizing 3804 the corresponding smoke cloud defined by the airborne particles.

[0217] At time t2, the application of energy to the tissue 3760 generates a smoke cloud 3752 at the surgical site, as shown by a second graphic 3892. The control circuit 132 may determine 3806 that the cloud density has exceeded a smoke cloud density threshold (e.g., represented by D3). Accordingly, the control circuit 132 adjusts 3808 the electrosurgical instrument energy duty cycle control parameter by decreasing 3872 it from E3 to E2. The control circuit 132 may select this adjustment because applying a lower level of energy to the tissue 3760 may generate less smoke. In response, the smoke cloud density begins to decrease 3864 at time t1.

[0218] At time t3, the smoke cloud 3752 has decreased in size but has not completely dissipated, as shown by the third graphic 3894. The control circuit 132 may determine 3806 that the cloud density is not decreasing at a fast enough rate or that some other characteristic of the smoke cloud has violated some other threshold. Thus, the control circuit 132 again adjusts 3808 the electrosurgical instrument energy duty cycle control parameter by decreasing 3874 from E2 to E1 to mitigate further smoke generation.

[0219] At time t4, the smoke cloud 3752 has nearly dissipated, as shown by the fourth graphic 3896. The control circuit 132 may determine 3806 that the smoke cloud has breached another threshold, such as a cloud density above a particular level (e.g., represented by D1) for longer than a threshold period (e.g., represented by t4). Thus, the control circuit 132 adjusts the smoke evacuator suction flow control parameter by increasing 3884 from Q2 to Q3 to completely remove the smoke particulates from the surgical site.

[0220] It should be noted that the implementation of process 3800 embodied by Figure 38 is provided for illustrative purposes and represents merely one possible implementation. In particular, different control parameters may be controlled by process 3800, different thresholds may be utilized, different smoke cloud characteristics may be tracked, etc. Thus, Figure 38 should not be construed in any way as limiting the process 3800 of Figure 37 or any other described systems and methods.

[0221] Example Clinical Uses The various surgical visualization systems disclosed herein may be used in one or more of the following clinical applications: The following clinical applications are non-exhaustive and merely exemplary applications for one or more of the various surgical visualization systems disclosed herein.

[0222] The surgical visualization system, as disclosed herein, can be used for many different types of procedures in different specialties, such as, for example, urology, gynecology, oncology, colorectal, thoracic surgery, bariatric / gastrocology, and hepatobiliary-pancreatic (HPB). For example, in urological surgery, such as prostatectomy, the urinary tract may be detected in fat or connective tissue, and / or nerves may be detected, for example, in fat. For example, in gynecologic oncology surgery, such as hysterectomy, and colorectal surgery, such as low anterior resection (LAR), the ureter may be detected, for example, in fat and / or connective tissue. For example, in thoracic surgery, such as lobectomy, blood vessels may be detected in lung or connective tissue, and / or nerves may be detected in connective tissue (e.g., esophagostomy). For bariatric surgery, blood vessels may be detected in fat. For example, in HPB procedures such as hepatectomy or pancreatic resection, blood vessels may be found in fat (extrahepatic), connective tissue (extrahepatic), and bile ducts may be found in parenchymal tissue (liver or pancreas).

[0223] In one example, a clinician may wish to remove an endometrial fibroid. From a preoperative magnetic resonance imaging (MRI) scan, the clinician may know that the endometrial fibroid is located on the surface of the intestine. Thus, the clinician may wish to know intraoperatively which tissues make up a portion of the intestine and which tissues make up a portion of the rectum. In such an example, a surgical visualization system, as disclosed herein, can indicate the different types of tissues (intestine vs. rectum) and communicate that information to the clinician via the imaging system. Additionally, the imaging system can determine and communicate the proximity of a surgical device to a selected tissue. In such an example, the surgical visualization system can improve procedure efficiency without significant complications.

[0224] In another example, a clinician (e.g., a gynecologist) may stay away from certain anatomical areas to avoid getting too close to critical structures, so that the clinician may not remove all of the endometriosis, for example. A surgical visualization system, as disclosed herein, may enable the gynecologist to mitigate the risk of getting too close to critical structures, so that the surgical device can get close enough to remove all of the endometriosis, improving patient outcomes (democratizing surgery). Such a system may enable the surgeon to "keep moving" during a surgical procedure, instead of repeatedly stopping and starting to identify areas to avoid, for example, especially during application of therapeutic energy, such as ultrasound or electrosurgical energy. In gynecological applications, the uterine arteries and ureters are important critical structures, and the system may be particularly useful for hysterectomies and endometrial surgery, given the presentation and / or thickness of the tissue involved.

[0225] In another example, a clinician may be at risk of dissecting a vessel in a location that is too close and may affect blood supply to lobes other than the targeted lobe. Additionally, anatomical differences between patients may result in dissection of a vessel (e.g., a branch vessel) that affects different lobes based on the particular patient. A surgical visualization system, as disclosed herein, can enable identification of the correct vessel at the desired location, thereby allowing the clinician to ensure that the proper anatomy is dissected. For example, the system can verify that the correct vessel is in the correct location, after which the clinician can safely divide the vessel.

[0226] In another example, a clinician may make multiple incisions before incising in the best location due to uncertainty about vascular anatomy. However, it is desirable to incise in the best location on the first go, as more incisions may increase the risk of bleeding. A surgical visualization system, as disclosed herein, can minimize the number of incisions by indicating the correct vessels and best locations for incision. For example, the ureter and cardinal ligament are densely packed and provide unique challenges during dissection. In such an example, minimizing the number of incisions may be particularly desirable.

[0227] In another example, a clinician (e.g., an oncology surgeon) removing cancerous tissue may want to know the identification of critical structures, the localization of the cancer, the staging of the cancer, and / or an assessment of the health of the tissue. Such information is beyond what the clinician sees with the "naked eye." A surgical visualization system, as disclosed herein, can determine and / or communicate such information to the clinician during surgery to enhance intraoperative decisions and improve surgical outcomes. In certain examples, the surgical visualization system can be compatible with minimally invasive surgery (MIS), open surgery, and / or robotic approaches, for example, using either an endoscope or an exoscope.

[0228] In another example, a clinician (e.g., an oncology surgeon) may wish to turn off one or more warnings regarding the proximity of a surgical tool to one or more critical structures to avoid being too conservative during surgery. In other examples, a clinician may wish to receive certain types of warnings, such as haptic feedback (e.g., vibration / buzzer) to indicate proximity and / or a "no-fly zone" to remain far enough away from one or more critical structures. The surgical visualization system, as disclosed herein, may provide adaptability based, for example, on the clinician's experience and / or the desired aggressiveness of the procedure. In such examples, the system provides a balance between "knowing too much" and "knowing enough" to anticipate and avoid critical structures. The surgical visualization system may assist in planning next steps during surgery.

[0229] Various aspects of the subject matter described herein are illustrated in the following numbered examples.

[0230] Example 1. A control system for a surgical system, the control system including an imaging system and a control circuit coupled to the imaging system. The imaging system includes an emitter configured to emit electromagnetic radiation (EMR), a polarizing filter configured to polarize the EMR emitted by the emitter, and an image sensor. The control circuit is configured to detect polarized EMR reflected from an airborne particle received by the image sensor, determine a particulate type of the airborne particulate whether it is a naturally occurring particulate or a synthetic particulate based on a reflectance characteristic of the polarized EMR, and adjust control parameters of the surgical system according to the determined particulate type.

[0231] Example 2. The control system of example 1, wherein the surgical system includes a smoke evacuator and the control parameters include at least one of a smoke evacuator flow rate, a fan level, or a motor level.

[0232] Example 3. The control system of Example 1, wherein the surgical system includes an electrosurgical instrument and the control parameters include at least one of a power level or an energy profile of the electrosurgical instrument.

[0233] Example 4. A control system as described in any one of Examples 1 to 3, wherein the airborne particulate is one of a plurality of airborne particulates, the image sensor includes a pixel array, and the control circuit is further configured to detect a change in detection of the plurality of airborne particulates according to the pixel array, and to characterize a movement of a surgical smoke cloud defined by the plurality of airborne particulates according to the detected change.

[0234] Example 5. The control system of Example 4, wherein the control circuit is further configured to determine a quantity of the plurality of airborne particles, to determine a vector of movement of the plurality of airborne particles, and to characterize the movement of the surgical cloud according to the determined quantity and the determined vector.

[0235] Example 6. The control system of any one of Examples 1 to 5, wherein the surgical hub comprises the control system.

[0236] Example 7. A control system for a surgical system, the control system including an imaging system and a control circuit coupled to the imaging system. The imaging system includes a multispectral electromagnetic radiation (EMR) source, a polarizing filter configured to polarize the EMR emitted by the multispectral EMR source, and an image sensor. The control circuit is configured to detect polarized EMR reflected from a particulate in the aerosol received by the image sensor, determine whether a particulate type of the particulate is a naturally occurring particulate or a synthetic particulate based on a reflectance characteristic of the polarized EMR, determine whether to adjust a first control parameter or a second control parameter according to the determined particulate type, and adjust the first control parameter or the second control parameter accordingly.

[0237] Example 8. The control system of Example 7, wherein the surgical system includes a smoke evacuator and the first control parameter includes at least one of a smoke evacuator flow rate, a fan level, or a motor level.

[0238] Example 9. The control system of Example 7, wherein the surgical system includes an electrosurgical instrument and the second control parameter includes at least one of a power level or an energy profile of the electrosurgical instrument.

[0239] Example 10. A control system described in any one of Examples 7 to 9, wherein the particulate is one of a plurality of suspended particulates, the image sensor includes a pixel array, and the control circuit is further configured to detect a change in detection of the plurality of particulates according to the pixel array, and to characterize the movement of the aerosol defined by the plurality of particulates according to the detected change.

[0240] Example 11. A control system as described in Example 10, wherein the control circuit is further configured to determine a quantity of the plurality of particles, to determine a vector of movement of the plurality of particles, and to characterize the movement of the aerosol according to the determined quantity and the determined vector.

[0241] Example 12. A method for controlling a surgical system, the surgical system including an imaging system, the imaging system including an emitter configured to emit electromagnetic radiation (EMR), a polarizing filter configured to polarize the EMR emitted by the emitter, and an image sensor. The method includes detecting polarized EMR reflected from an airborne particle received by the image sensor, determining whether a particle type of the airborne particle is a naturally occurring particle or a synthetic particle based on a reflectance characteristic of the polarized EMR, and adjusting a control parameter of the surgical system according to the determined particle type.

[0242] Example 13. The method of Example 12, wherein the surgical system further includes a smoke evacuator, and the control parameters include at least one of a smoke evacuator flow rate, a fan level, or a motor level.

[0243] Example 14. The method of Example 12, wherein the surgical system further includes an electrosurgical instrument, and the control parameters include at least one of a power level or an energy profile of the electrosurgical instrument.

[0244] Example 15. A method as described in any one of Examples 12 to 14, wherein the airborne particulate is one of a plurality of airborne particulates, the image sensor includes a pixel array, and the method further includes detecting a change in detection of the plurality of airborne particulates according to the pixel array, and characterizing a movement of a surgical smoke cloud defined by the plurality of airborne particulates according to the detected change.

[0245] Example 16. The method of Example 15, further comprising determining a quantity of a plurality of airborne particles, determining a vector of movement of the plurality of airborne particles, and characterizing the movement of the surgical cloud according to the determined quantity and the determined vector.

[0246] Example 17. The method of any one of Examples 12 to 16, wherein the surgical hub constitutes a surgical system.

[0247] While several embodiments have been shown and described, it is not the intention of the applicant to restrict or limit the scope of the appended claims to such details. Numerous modifications, variations, changes, substitutions, combinations, and equivalents of these embodiments can be implemented and will occur to those skilled in the art without departing from the scope of the present disclosure. Moreover, the structure of each element associated with the described embodiments can be alternatively described as a means for providing the function performed by that element. Also, although materials are disclosed with respect to specific components, other materials may be used. It is therefore to be understood that the above description and the appended claims are intended to cover all such modifications, combinations, and variations as falling within the scope of the disclosed embodiments. The appended claims are intended to cover all such modifications, variations, changes, substitutions, modifications, and equivalents.

[0248] The above detailed description has set forth various aspects of the apparatus and / or processes via the use of block diagrams, flow diagrams and / or examples. To the extent that such block diagrams, flow diagrams and / or examples include one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation included in such block diagrams, flow diagrams and / or examples can be individually and / or collectively implemented by various hardware, software, firmware, or virtually any combination thereof. Those skilled in the art will understand that some aspects of the embodiments disclosed herein may be equivalently implemented in whole or in part on an integrated circuit as one or more computer programs running on one or more computers (e.g., one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., one or more programs running on one or more microprocessors), as firmware, or as substantially any combination thereof, and that designing circuitry and / or writing software and / or firmware code is within the skill of one of ordinary skill in the art in view of this disclosure. In addition, those skilled in the art will understand that the subject mechanisms described herein may be distributed as one or more program products in a variety of forms, and that the specific forms of the subject matter described herein apply regardless of the particular type of signal-bearing medium used to actually effect the distribution.

[0249] The instructions used to program the logic to implement the various disclosed aspects may be stored in a system memory such as dynamic random access memory (DRAM), cache, flash memory, or other storage. Additionally, the instructions may be distributed over a network or by other computer-readable media. Machine-readable media may therefore include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to floppy diskettes, optical disks, compact disks, read-only memories (CD-ROMs), as well as magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memories, or tangible machine-readable storage used to transmit information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Non-transitory computer-readable media may therefore include any type of tangible machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0250] The term "control circuitry" as used in any aspect of the present specification may refer to, for example, hardwired circuitry, programmable circuitry (e.g., a computer processor including one or more individual instruction processing cores, a processing unit, a processor, a microcontroller, a microcontroller unit, a controller, a digital signal processor (DSP), a programmable logic device (PLD), a programmable logic array (PLA), or a field programmable gate array (FPGA)), a state machine circuit, firmware that stores instructions executed by the programmable circuit, and any combination thereof. The control circuitry may be embodied, collectively or individually, as circuits that form part of a larger system, such as, for example, an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc. Thus, as used herein, "control circuitry" includes, but is not limited to, electrical circuitry having at least one discrete electrical circuit, electrical circuitry having at least one integrated circuit, electrical circuitry having at least one application specific integrated circuit, electrical circuitry forming a general purpose computing device configured by a computer program (e.g., a general purpose computer configured by a computer program that at least partially executes the processes and / or devices described herein, or a microprocessor configured by a computer program that at least partially executes the processes and / or devices described herein), electrical circuitry forming a memory device (e.g., a form of random access memory) and / or electrical circuitry forming a communications device (e.g., a modem, a communications switch, or an optical-electrical facility). Those skilled in the art will recognize that the subject matter described herein may be implemented in analog or digital form, or some combination thereof.

[0251] As used in any aspect herein, the term "logic" may refer to an application, software, firmware, and / or circuitry configured to perform any of the operations described above. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets, and / or hard-coded (e.g., non-volatile) data in a memory device.

[0252] As used in any aspect of this specification, the terms "component," "system," "module," etc. may refer to a computer-related entity that is either hardware, a combination of hardware and software, software, or software in execution.

[0253] As used in any aspect of this specification, an "algorithm" refers to a self-consistent sequence of steps leading to a desired result, and the "steps" refer to manipulations of physical quantities and / or logical states, which may, but need not, take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is common practice to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. These and similar terms may be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities and / or states.

[0254] The network may include a packet-switched network. The communication devices may communicate with each other using a selected packet-switched network communication protocol. One exemplary communication protocol may include an Ethernet communication protocol that may enable communication using Transmission Control Protocol / Internet Protocol (TCP / IP). The Ethernet protocol may conform to or be compatible with the Ethernet standard entitled "IEEE 802.3 Standard" published in December 2008 by the Institute of Electrical and Electronics Engineers (IEEE), and / or later versions of this standard. Alternatively or additionally, the communication devices may communicate with each other using an X.25 communication protocol. The X.25 communication protocol may conform to or be compatible with standards promulgated by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T). Alternatively or additionally, the communication devices may communicate with each other using a Frame Relay communication protocol. The frame relay communication protocol may conform to or be compatible with standards promulgated by the Consultative Committee for International Telegraph and Telephone (CCITT) and / or the American National Standards Institute (ANSI). Alternatively or additionally, the transceivers may be capable of communicating with each other using an Asynchronous Transfer Mode (ATM) communication protocol. The ATM communication protocol may conform to or be compatible with the ATM standard published in August 2001 by the ATM Forum entitled "ATM-MPLS Network Interworking 2.0" and / or later versions of this standard. Of course, different and / or later developed connection-oriented network communication protocols are equally contemplated herein.

[0255] Unless expressly specified otherwise, as will be apparent from the foregoing disclosure, discussions throughout the foregoing disclosure using terms such as "processing," "computing," "calculating," "determining," "displaying," and the like will be understood to refer to the actions and processing of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the computer system's registers and memory into other data that are similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0256] One or more components may be referred to herein as being "configured to," "configurable to," "operable / operative to," "adapted / adaptable," "able to," "conformable / conformed to," etc. Those skilled in the art will understand that "configured to" may generally encompass active components and / or inactive components and / or standby components, unless the context requires otherwise.

[0257] The terms "proximal" and "distal" are used herein with reference to a clinician manipulating a handle portion of a surgical instrument. The term "proximal" refers to the portion closest to the clinician and the term "distal" refers to the portion located away from the clinician. It will be further understood that for convenience and clarity, spatial terms such as "vertical," "horizontal," "up," and "down" may be used herein with respect to the drawings. However, surgical instruments are used in many orientations and positions, and these terms are not intended to be limiting and / or absolute.

[0258] Those skilled in the art will understand that the terms used herein in general, and in the appended claims in particular (e.g., the body of the appended claims) are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including but not limited to," the term "having" should be interpreted as "having at least," the term "includes" should be interpreted as "includes but is not limited to," etc.). Moreover, those skilled in the art will understand that where a specific number is intended in an introduced claim recitation, such intention will be clearly recited in the claim, and in the absence of such recitation, no such intention exists. For example, as an aid to understanding, the appended claims below may include the introductory phrases "at least one" and "one or more" to introduce the claim recitation. However, the use of such phrases should not be construed as implying that when a claim recitation is introduced by the indefinite article "a" or "an," any particular claim containing such an introduced claim recitation is limited to claims containing only one such recitation, even if the same claim contains an introductory phrase such as "one or more" or "at least one" and the indefinite article "a" or "an" (e.g., "a" and / or "an" should normally be construed to mean "at least one" or "one or more"). The same applies when a definite article is used to introduce a claim recitation.

[0259] In addition, those of skill in the art will recognize that even when a particular number is specified in an introduced claim recitation, such recitation should typically be interpreted to mean at least the recited number (e.g., a recitation of simply "two items" without other qualifiers typically means at least two items, or two or more items). Furthermore, when a notation similar to "at least one of A, B, and C, etc." is used, such syntax is generally intended in the sense that a person of skill in the art would understand the notation (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C, etc.). When a notation similar to "at least one of A, B, or C, etc." is used, generally such syntax is intended in the sense that one of ordinary skill in the art would understand the notation (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, both A and B, both A and C, both B and C, and / or all of A, B, and C, etc.). Furthermore, one of ordinary skill in the art will understand that any disjunctive word and / or phrase expressing two or more alternative terms should typically be understood to contemplate the possibility of including one of those terms, either of those terms, or both of those terms, unless the context requires otherwise, whether in the specification, claims, or drawings. For example, the phrase "A or B" will typically be understood to include the possibilities of "A" or "B" or "A and B."

[0260] With respect to the appended claims, one of ordinary skill in the art will appreciate that the recited operations herein may generally be performed in any order. Also, while flow diagrams of various operations are shown in a sequence, it should be understood that the various operations may be performed in orders other than those shown, or may be performed simultaneously. Examples of such alternative orderings may include overlapping, interleaving, interrupting, reordering, incremental, preliminary, additional, simultaneous, reverse, or other different orderings, unless the context requires otherwise. Moreover, terms such as "responsive to," "related to," or other past tense adjectives are not generally intended to exclude such variations, unless the context requires otherwise.

[0261] It is worth noting that any reference to "one embodiment," "an embodiment," "an example," "an example," or the like means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment," "in an embodiment," "in an example," and "in one example" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0262] Any patent application, patent, non-patent publication, or other disclosure material referenced herein and / or listed in any Application Data Sheet is incorporated herein by reference to the extent that the incorporated material is not inconsistent with this specification. As such, and to the extent necessary, the disclosure material explicitly set forth in this specification shall take precedence over any conflicting statements incorporated herein by reference. Any material, or portions thereof, that is referred to as being incorporated herein by reference but that conflicts with current definitions, opinions, or other disclosure material set forth herein shall be incorporated only to the extent that no conflict arises between the incorporated material and the current disclosure material.

[0263] In summary, many benefits have been described that result from using the concepts described herein. The above description of one or more embodiments has been presented for purposes of illustration and description. It is not intended to be exhaustive or to be limited to the precise form disclosed. Modifications or variations are possible in light of the above teachings. The one or more embodiments have been selected and described to illustrate the principles and practical applications, thereby enabling those skilled in the art to utilize the various embodiments, with various modifications, as appropriate for the particular use contemplated. It is intended that the claims presented herewith define the overall scope.

[0264] [Embodiment] (1) A control system for a surgical system, comprising: 1. An imaging system, comprising: an emitter configured to emit electromagnetic radiation (EMR); a polarizing filter configured to polarize the EMR emitted by the emitter; and an imaging system including an image sensor; a control circuit coupled to the imaging system, detecting the polarized EMR reflected from airborne particles received by the image sensor; determining whether the airborne particulate is a naturally occurring particulate or a synthetic particulate based on the reflectance characteristics of the polarized EMR; and and a control circuit configured to adjust a control parameter of the surgical system according to the determined particulate type. (2) the surgical system includes a smoke evacuator; 2. The control system of embodiment 1, wherein the control parameters include at least one of the smoke exhauster flow rate, fan level, or motor level. (3) the surgical system includes an electrosurgical instrument; 2. The control system of claim 1, wherein the control parameters include at least one of a power level or an energy profile of the electrosurgical instrument. (4) the airborne particulate is one of a plurality of airborne particulates; the image sensor includes a pixel array; The control circuit includes: detecting a change in detection of the plurality of airborne particles according to the pixel array; and 2. The control system of claim 1, further configured to characterize the movement of a surgical smoke cloud defined by the plurality of airborne particles in accordance with the detected changes. (5) The control circuit determining a quantity of the plurality of airborne particles; determining a vector of motion of said plurality of airborne particles; and 5. The control system of claim 4, further configured to characterize the movement of the surgical cloud according to the determined quantity and the determined vector.

[0265] (6) A control system as described in embodiment 1, wherein a surgical hub constitutes the control system. (7) A control system for a surgical system, comprising: 1. An imaging system, comprising: A multispectral electromagnetic radiation (EMR) source; a polarizing filter configured to polarize the EMR emitted by the multispectral EMR source; and an imaging system including an image sensor; a control circuit coupled to the imaging system, detecting the polarized EMR reflected from particles in an aerosol received by the image sensor; determining whether the particle type of the particle is a naturally occurring particle or a synthetic particle based on the reflectance characteristics of the polarized EMR; determining whether to adjust a first control parameter or a second control parameter according to the determined particulate type; and and a control circuit configured to adjust the first control parameter or the second control parameter accordingly. (8) the surgical system includes a smoke evacuator; A control system as described in embodiment 7, wherein the first control parameter includes at least one of a flow rate, a fan level, or a motor level of the smoke evacuation device. (9) the surgical system includes an electrosurgical instrument; A control system as described in embodiment 7, wherein the second control parameter includes at least one of a power level or an energy profile of the electrosurgical instrument. (10) The microparticle is one of a plurality of microparticles; the image sensor includes a pixel array; The control circuit includes: detecting a change in detection of the plurality of particles according to the pixel array; and 8. The control system of claim 7, further configured to characterize the movement of the aerosol defined by the plurality of particles according to the detected changes.

[0266] (11) The control circuit determining a quantity of the plurality of particulates; determining vectors of motion of the plurality of particles; and 11. The control system of embodiment 10, further configured to characterize the movement of the aerosol according to the determined quantity and the determined vector. (12) A method for controlling a surgical system, the surgical system including an imaging system, the imaging system including an emitter configured to emit electromagnetic radiation (EMR), a polarizing filter configured to polarize the EMR emitted by the emitter, and an image sensor, the method including: detecting the polarized EMR reflected from airborne particles received by the image sensor; and determining a particle type of the airborne particles as naturally occurring particles or synthetic particles based on the reflectance characteristics of the polarized EMR; and adjusting a control parameter of the surgical system according to the determined particulate type. (13) The surgical system further comprises a smoke evacuator; 13. The method of claim 12, wherein the control parameters include at least one of the smoke evacuator flow rate, fan level, or motor level. (14) The surgical system further includes an electrosurgical instrument; 13. The method of claim 12, wherein the control parameters include at least one of a power level or an energy profile of the electrosurgical instrument. (15) The airborne particulate is one of a plurality of airborne particulates; the image sensor includes a pixel array; The method comprises: detecting a change in detection of the plurality of airborne particles according to the pixel array; and 13. The method of claim 12, further comprising: characterizing a movement of a surgical smoke cloud defined by the plurality of airborne particulates according to the detected changes.

[0267] (16) The method further comprises: determining a quantity of the plurality of airborne particles; determining vectors of motion of the plurality of airborne particles; 16. The method of claim 15, further comprising: characterizing the movement of the surgical cloud according to the determined quantity and the determined vector. (17) The method of embodiment 12, wherein a surgical hub constitutes the surgical system.

Claims

1. 1. A control system for a surgical system, comprising:

1. An imaging system, comprising: an emitter configured to emit electromagnetic radiation (EMR); a polarizing filter configured to polarize the EMR emitted by the emitter; and an imaging system including an image sensor; a control circuit coupled to the imaging system, detecting the polarized EMR reflected from airborne particles received by the image sensor; determining whether the airborne particulate is a naturally occurring particulate or a synthetic particulate based on the reflectance characteristics of the polarized EMR; and and a control circuit configured to adjust a control parameter of the surgical system according to the determined particulate type.

2. the surgical system includes a smoke evacuator; 2. The control system of claim 1, wherein the control parameters include at least one of a flow rate of the smoke evacuator, a power level delivered to a fan of the smoke evacuator, or a power level delivered to a smoke evacuator motor.

3. The surgical system includes an electrosurgical instrument; The control system of claim 1 , wherein the control parameter includes a power level of the electrosurgical instrument.

4. the airborne particulate is one of a plurality of airborne particulates; the image sensor includes a pixel array; The control circuit includes: detecting a change in detection of the plurality of airborne particles according to the pixel array; and The control system of claim 1 , further configured to characterize a movement of a surgical smoke cloud defined by the plurality of airborne particulates in accordance with the detected changes.

5. The control circuit includes: determining a quantity of the plurality of airborne particles; determining a vector of motion of said plurality of airborne particles; and The control system of claim 4 , further configured to characterize the movement of the surgical smoke cloud according to the determined quantity and the determined vector.

6. The control system of claim 1 , wherein a surgical hub comprises the control system.

7. 1. A control system for a surgical system, comprising:

1. An imaging system, comprising: a multispectral electromagnetic radiation (EMR) source; a polarizing filter configured to polarize the EMR emitted by the multispectral EMR source; an imaging system including an image sensor; a control circuit coupled to the imaging system, detecting the polarized EMR reflected from particles in an aerosol received by the image sensor; determining a particle type of the particle as being a naturally occurring particle or a synthetic particle based on a reflectance characteristic of the polarized EMR; determining whether to adjust a first control parameter or a second control parameter according to the determined particulate type; and and a control circuit configured to adjust the first control parameter or the second control parameter accordingly.

8. the surgical system includes a smoke evacuator; 8. The control system of claim 7, wherein the first control parameter comprises at least one of a flow rate of the smoke evacuator, a power level delivered to a fan of the smoke evacuator, or a power level delivered to a smoke evacuator motor.

9. The surgical system includes an electrosurgical instrument; The control system of claim 7 , wherein the second control parameter includes a power level of the electrosurgical instrument.

10. the particulate is one of a plurality of particulates; the image sensor includes a pixel array; The control circuit includes: detecting a change in detection of the plurality of particles according to the pixel array; and The control system of claim 7 , further configured to characterize a motion of the aerosol defined by the plurality of particulates according to the detected changes.

11. The control circuit includes: determining a quantity of the plurality of particulates; determining vectors of motion of the plurality of particles; and The control system of claim 10 , further configured to characterize the movement of the aerosol according to the determined amount and the determined vector.

12. 1. A method for controlling a surgical system, the surgical system including an imaging system, the imaging system including an emitter configured to emit electromagnetic radiation (EMR), a polarizing filter configured to polarize the EMR emitted by the emitter, and an image sensor, the method comprising: detecting the polarized EMR reflected from airborne particles received by the image sensor; determining a particle type of the airborne particles as naturally occurring particles or synthetic particles based on a reflectance characteristic of the polarized EMR; and adjusting a control parameter of the surgical system according to the determined particulate type.

13. The surgical system further comprises a smoke evacuator; 13. The method of claim 12, wherein the control parameters include at least one of a flow rate of the smoke evacuator, a power level delivered to a fan of the smoke evacuator, or a power level delivered to a smoke evacuator motor.

14. The surgical system further includes an electrosurgical instrument; The method of claim 12, wherein the control parameter includes a power level of the electrosurgical instrument.

15. the airborne particulate is one of a plurality of airborne particulates; the image sensor includes a pixel array; The method comprises: detecting a change in detection of the plurality of airborne particles according to the pixel array; and 13. The method of claim 12, further comprising: characterizing a movement of a surgical smoke cloud defined by the plurality of airborne particles according to the detected changes.

16. The method comprises: determining a quantity of the plurality of airborne particles; determining vectors of motion of the plurality of airborne particles; 16. The method of claim 15, further comprising: characterizing the movement of the surgical smoke cloud according to the determined quantity and the determined vector.

17. The method of claim 12 , wherein a surgical hub comprises the surgical system.

Images